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Abstract 

As  government  and  industry  becomes  subject  to  a  wider  range  of  technology  initiatives, 
science  and  technology  (S&T)  research  project  leadership  recognizes  the  need  to 
incorporate  more  systems  engineering  (SE)  rigor  into  their  projects.  The  objective  of  this 
research  is  to  develop  a  tailorable  systems  engineering  framework  for  S&T  project 
planning,  execution,  assessment  and  transition.  The  key  deliverable  is  an  Excel-based 
tool  instantiating  the  SE  framework  for  a  wide  range  of  S&T  projects  in  technology 
development  organizations.  It  includes  a  report  with  tailored  methods  based  on 
programmatic  discriminants. 

To  develop  this  framework,  a  comprehensive  understanding  of  SE  principles  is  applied  to 
several  case  studies  across  government  and  supporting  industry-sponsored  S&T 
activities.  This  research  followed  a  six-step  approach:  (1)  Literature  Review;  (2) 
Formulate  Taxonomy;  (3)  Prepare  Data  Gathering  Approach;  (4)  Review  Case  Studies; 

(5)  Develop  Tailorable  SE  Framework  for  Technology  Development  and  Transition;  and 

(6)  Validate  Framework. 

The  framework  allows  S&T  project  leaders  and  engineers  to  customize  a  recommended 
set  of  SE  processes,  methods  and  tools  for  their  specific  project  type,  size,  maturity, 
budget,  and  integration  level.  Recommendations  for  SE  methods  are  made  at  a  summary 
level,  with  additional  details  available  for  desired  activities.  References  to  established  SE 
documentation  is  also  included  for  further  investigation  of  appropriate  SE  techniques. 
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A  TAILORED  SYSTEMS  ENGINEERING  FRAMEWORK 


FOR  SCIENCE  AND  TECHNOLOGY  PROJECTS 

I.  Introduction 

The  Air  Force  Research  Laboratory  (AFRL)  has  a  long  history  of  developing  advanced 
technologies  that  ultimately  deliver  effects  to  the  battlefield,  whether  in  preparation, 
planning,  or  combat  force.  As  research  and  development  (R&D)  dollars  become  subject 
to  a  wider  range  of  technology  initiatives,  AFRL  leadership  recognizes  the  need  to 
incorporate  more  systems  engineering  (SE)  rigor  into  their  projects.  This  chapter 
addresses  the  objective,  scope,  and  approach  of  a  research  effort  to  help  AFRL 
implement  tailored  SE  to  Science  and  Technology  (S&T)  projects. 

Research  Objective 

The  objective  of  this  research  thesis  project  is  to  develop  a  tailorable  systems  engineering 
framework  for  science  and  technology  development  planning,  project  planning, 
execution,  assessment  and  transition.  It  provides  recommendations  to  validate  or 
improve  existing  SE  practices  within  AFRL.  The  key  deliverable  is  an  SE  framework, 
which  includes  a  thesis  with  tailored  methods  and  tools  based  on  user-selected  program 
discriminants.  If  implemented,  it  will  facilitate  use  of  SE  principles  by  technology 
developers,  project  or  program  managers,  decision  makers,  scientists,  and  engineers. 

Research  Scope 

The  team  established  several  guidelines  to  help  focus  the  research  and  establish  a  useful 
yet  manageable  scope.  First,  the  ultimate  goal  is  to  deliver  a  product  that  will  actually  be 
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utilized  by  the  sponsor  organization.  The  most  realistic  opportunity  for  this  to  occur  is  to 
deliver  a  framework  for  tailoring  SE  activities  -  something  that  can  assist  S&T  managers 
implement  the  appropriate  level  of  SE  rigor  for  their  specific  project.  One  of  the  biggest 
impediments  of  SE  application  by  the  S&T  community  is  the  mindset  that  “big  SE 
doesn’t  apply  to  my  specific  project.”  The  detail  of  the  SE  tailoring  tool  is  reflective  of 
the  research  team’s  desire  to  overcome  this  mental  obstacle  by  providing  an  easily 
navigated  map  indicating  appropriate  levels  of  SE  rigor  for  the  current  state  of  a  project. 

Next,  the  research  focuses  on  the  deliberate  and  thoughtful  application  of  SE  processes, 
methods  and  tools  to  the  S&T  community  rather  than  the  larger  systems  acquisition 
genre.  S&T  projects  may  not  meet  milestone  reviews,  might  not  get  detailed 
requirements  flowed  down  from  an  operational  user,  and  may  not  even  be  intended  for 
actual  use  as  developed.  Basic  research  projects  don’t  require  the  same  depth  of 
architecture  definition  that  is  critical  to  major  weapons  system  development  programs, 
nor  do  they  necessarily  have  the  resources  to  establish  a  comprehensive  SE  regimen. 
However,  early  SE  is  critical  for  subsequent  transition  of  S&T  products,  whether  to  a 
larger  integration  effort  or  to  the  field  as  an  Advanced  Concept  Technology  Demonstrator 
(ACTD).  Limiting  the  recommendation  for  SE  implementation  to  only  those  activities 
required  to  ensure  smart  transition  down  the  road  helps  S&T  projects  deliver  better 
products  without  all  of  the  resource-consuming  SE  rigor  demanded  of  larger  system 
acquisition  activities. 
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Finally,  this  report  takes  a  comprehensive  look  at  candidate  SE  processes,  methods  and 
tools  available  to  S&T  projects  based  on  academic  and  practical  research.  The  authors  of 
the  report  possess  a  knowledge  base  of  SE  principles  based  on  dedicated  coursework  in 
an  accredited  academic  program  and  are  guided  by  multiple  doctoral-level  SE  experts. 
This  provides  a  foundation  of  academic  SE  insight  which  is  bolstered  by  additional 
research  into  DoD  and  industry  SE  practices.  The  comprehensive  understanding  of  SE 
principles  is  then  applied  to  several  case  studies  within  DoD  and  in  commercial  research 
and  development  activities.  The  across-the-board  look  at  SE  applications  allows 
incorporation  of  best  practices  by  organizations  not  constrained  by  established  DoD 
processes  and  whose  S&T  successes  are  the  lifeblood  of  future  capabilities. 

During  the  initial  thesis  project  planning  activities,  AFRL  stakeholders  made  several 
suggestions  as  to  how  to  best  improve  the  SE  application  within  the  organization.  Most 
of  those  who  provided  input  claimed  that  culture  was  the  primary  inhibitor  of  true  SE 
success.  Others  raised  the  issue  that  the  division  of  responsibility  for  SE  between 
government  and  contract  personnel  was  a  significant  issue.  While  these  observations  are 
by  no  means  inaccurate  or  unimportant,  they  are  not  the  primary  focus  of  this  research. 
Rather,  the  intent  is  for  the  thesis  deliverables  to  make  desired  SE  benefits  more 
tailorable,  efficient  and  attainable. 

Approach 

This  research  followed  a  six-step  approach  that  will  be  described  in  more  detail  in 
Chapters  II  -  IV.  These  steps  are: 

1 .  Review  Literature  (Chapter  II) 
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The  research  team  conducted  an  extensive  literature  review  including  Air 
Force  policy,  guidance,  and  best  practices  at  all  levels  (DoD,  Air  Force, 
Air  Force  Material  Command,  and  Air  Force  Research  Laboratory).  The 
team  reviewed  existing  S&T  project  taxonomies  and  processes  within 
AFRL,  and  conducted  interviews  with  relevant  stakeholders.  The  team 
also  reviewed  systems  engineering  community  publications  from 
commercial,  academic,  and  professional  sources.  Comparisons  were  made 
with  previous  studies  that  analyzed  the  subject  of  systems  engineering  in 
an  S&T  environment,  ensuring  the  overlap  with  prior  efforts  did  not 
render  this  activity  redundant.  The  goal  of  this  phase  was  to  develop  a 
current  knowledge  base  with  regard  to  theory,  policy,  guidance,  best 
practices  and  shortfalls  of  SE  application  within  S&T  organizations. 

2.  Formulate  Taxonomy  (Chapter  III) 

The  research  team  synthesized  existing  taxonomies  and  processes  in  order 
to  tailor  relevant  SE  processes,  methods  and  tools  to  a  wide  range  of  S&T 
projects.  Logical  groupings  of  SE  activities  were  defined.  The  team 
standardized  a  reference  frame  for  S&T  projects  at  various  levels  of 
maturity,  given  existing  project  taxonomies.  Every  effort  was  made  to 
accommodate  and/or  relate  terms  to  existing  taxonomies.  The  goal  of  this 
phase  was  to  establish  common,  manageable  definitions  of  AFRL  S&T 
project  types  and  SE  principles. 
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3.  Prepare  Data  Gathering  Approach  (Chapter  III) 


The  research  defined  information  needs  based  on  where  a  technology 
development  effort  fit  within  the  taxonomy.  Information  needs  were 
defined  to  support  decision  making  based  on  project  objectives.  The  team 
also  identified  commonly  used  tools  to  accomplish  specific  SE  activities. 
The  goal  of  this  phase  was  to  establish  the  SE  taxonomy,  define  questions 
to  be  asked  and  information  to  be  gathered  during  the  case  study 
investigations. 

4.  Review  Case  Studies/Examples  (Chapter  IV) 

The  research  team  examined  projects  and  case  studies  to  report  on 
successful  application  of  SE  methods  and  gaps  in  SE  execution.  The 
review  included  active  and  historic  AFRL  projects  and  commercial 
projects.  The  goal  of  this  phase  was  to  extract  lessons  learned  from  a 
broad  cross-section  of  S&T  projects  and  make  direct  application  to 
improve  the  SE  framework  delivered  to  AFRL. 

5.  Develop  Tailorable  SE  Framework  for  Technology  Development  and 
Transition  (Chapter  IV) 

The  research  team  analyzed  lessons  learned  and  developed  a  tailorable 
approach  for  applying  SE  within  an  S&T  organization.  The  lessons 
learned  from  the  research  and  case  studies  provided  the  basis  for 


5 


recommended  SE  practices  and  strengthened  the  tailoring  of  SE  processes, 
methods  and  tools  within  the  S&T  framework  for  a  given  project  state. 

The  goal  of  this  phase  was  to  provide  a  framework  and  guidance  for 
interested  parties  to  add  SE  value  to  future  S&T  projects. 

6.  Validate  Framework  (Chapter  IV) 

This  research  obtained  feedback  from  the  stakeholders  as  to  the 
applicability  of  the  SE  framework.  Interested  and  knowledgeable  parties 
conducted  an  independent  evaluation  of  the  framework  by  evaluating 
typical  and  random  sets  of  programmatic  discriminants  and  incorporating 
recommended  changes.  The  goal  of  this  phase  was  to  deliver  a  tailorable 
SE  framework  for  S&T  development  planning,  project  planning, 
execution,  assessment  and  transition. 

The  specific  research  is  detailed  in  the  remainder  of  this  report.  The  background  of  SE 
within  AFRL  as  well  as  a  thorough  literature  review  is  described  in  Chapter  II.  Chapter 
III  defines  the  research  methodology,  including  descriptions  of  the  taxonomies  and 
information  needs,  as  well  as  the  process  for  conducting  the  case  studies  and  validation  of 
the  SE  framework.  Relevant  case  study  reviews  and  application  of  the  extracted  lessons 
learned  to  the  tailorable  SE  framework  development  and  validation  comprise  Chapter  IV. 
Chapter  V  discusses  the  final  results  and  recommendations  of  the  research. 
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II.  Background 


This  chapter  addresses  the  resources  utilized  by  the  team  for  the  information  gathering 
phase,  to  include  previous  efforts  to  study  and  improve  early  application  of  SE,  published 
policies,  and  documents  from  DoD,  professional  and  academic  communities.  The 
information  provides  a  baseline  as  to  what  has  been  done,  and  opens  questions  as  to  what 
more  could  be  done.  The  SE  history  within  AFRL,  their  current  practices,  and  the 
obstacles  to  successful  SE  implementation  within  AFRL  all  provide  the  impetus  for  the 
SE  tailoring  framework  developed  by  this  thesis. 

SE  History  within  AFRL 

Integrated  Process  and  Project  Development  (IPPD)  is  a  structured  SE  process  including 
management  principles,  design  philosophy,  methodology,  and  tools  which  was  formally 
instituted  within  AFRL  in  2000  [IPPD,  2002:  iv] .  A  primary  assertion  of  the  IPPD 
document  with  respect  to  SE  is  “The  finished  dish  might  be  new,  but  the  ingredients  have 
been  on  the  store  shelves  all  along”  [IPPD,  2002:  2],  IPPD  aims  to  increase  the  amount 
of  integration  and  SE  activity  by  focusing  on  requirements,  exit  criteria,  technology 
alternatives,  and  decision  analysis.  The  IPPD  approach  proved  effective  in  industry  and 
is  also  adaptable  to  S&T  to  provide  a  map  to  implement  SE  methods  for  development 
projects. 

Two  significant  reviews  of  SE  application  within  AFRL  lay  out  prior  successes  and 
opportunities  for  improvements  that  are  being  realized  today.  In  2004,  the  Air  Force 
Institute  of  Technology’s  report  on  “Technology  Transition  and  Program  Formulation  in 
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AFRL”  called  for  “integration  of  technologies  between  technical  directorates  and  the 
need  for  a  firm  grasp  on  system  engineering  principles.”  The  two  initial 
recommendations  from  this  report  are:  1)  improve  the  application  of  SE  principles,  and  2) 
change  the  culture  at  AFRL  [Coglitore  et  al,  2004:  2],  Following  the  implementation  of 
IPPD,  General  Dynamics  produced  the  Transformational  Activities  in  Systems 
Engineering  (TASE)  Report  to  evaluate  SE  practices,  to  include  IPPD  implementation 
and  effectiveness,  for  AFRL/XP  in  May  2007.  One  of  the  recommendations  was,  “AFRL 
should  use  the  Defense  Acquisition  Guidebook  (Chapter  4)  as  a  framework  for  improving 
its  systems  engineering  guidance  because  it  is  complete  from  a  process  viewpoint  and  is 
supported  by  DoD”  [TASE,  2006:  1]. 

Current  AFRL  SE  Practices 

Subsequent  to  the  TASE  report,  AFRL  implemented  two  active  governing  instructions 
for  SE  policy.  AFRL  Instruction  61-104,  “Science  and  Technology  Systems 
Engineering”  provides  direction  to  ensure  that  SE  is  implemented  on  all  S&T  programs, 
although  the  application  of  SE  to  basic  research  programs  is  optional  to  the  director  of 
each  technology  directorate  and  the  Air  Force  Office  of  Scientific  Research  (AFOSR) 
[S&T  SE,  2008].  The  instruction  states  that  the  level  of  SE  effort  is  to  be  tailored  to  the 
needs  of  the  individual  S&T  activity  and  customer  expectations,  and  provides  eight  SE 
key  questions  to  assess  programs.  The  instruction  also  evaluates  the  16  Defense 
Acquisition  Guidebook  (DAG)  processes  (8  Technical  Processes,  8  Technical 
Management  Processes)  specific  to  the  S&T  activities,  by  re-writing  the  DAG  processes 
in  AFRL  language  and  stating  the  importance  of  each  [DAG,  2004].  This  research  took 
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the  importance  of  tailoring  and  addressing  all  16  DAG  processes  into  account.  AFRL 
Instruction  61-202,  “AFRL  Laboratory  Management  Review  (LMR)  Process”  provides  a 
logical  approach  for  laboratory  reviews,  to  include  an  extensive  listing  of  questions  to 
assess  each  area  of  a  project  (technical,  financial,  schedule,  contracting,  deliverables, 
manning,  and  testing)  [LMR,  2005].  These  current  practices  introduce  SE  at  the 
conceptual  level  but  do  not  proscribe  detailed  or  tailored  SE  regimens  for  all  S&T 
projects. 

A  major  AFRL  initiative  started  in  2006  is  the  use  of  Focused  Long-Term  Challenges 
(FLTCs)  to  increase  the  S&T  integration  level  across  AFRL’s  Technology  Directorates. 
S&T  project  integration  across  AFRL  directorates  is  required  to  meet  capability 
objectives  established  by  combatant,  operational,  and  development  commands.  The 
integration  challenge  is  well-known  to  AFRL,  as  several  studies,  initiatives  and  policies 
(including  the  TASE  Report,  an  AFIT  research  effort,  and  AFRLI  61-104)  demanded 
stronger  integration  efforts  in  order  to  transition  S&T  project  successes.  The  FLTC 
initiative  organizes  the  majority  of  AFRL  projects  into  one  of  eight  Challenge  categories, 
then  further  subdivides  them  into  Problems,  Capability  Concepts,  Products,  and 
Programs.  FLTCs  are  designed  to  produce  integrated  technology  challenge  baselines, 
taxonomies,  and  roadmaps  to  show  how  groups  of  separately  managed  products  will 
deliver  integrated  capabilities  [FLTC  Briefing,  2006].  In  Fall  2008,  the  FLTCs  were 
evaluated  by  an  Independent  Review  Team  (IRT)  headed  by  the  Director  of  the  Air 
Force’s  Center  for  Systems  Engineering.  An  interview  with  the  Director  provided 
several  recommendations  about  good  project  case  study  candidates.  Some  concerns  were 
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raised  about  FLTC’s  cross-directorate  integration  success,  including  lack  of 
demonstrations  at  the  integrated  systems  level,  lack  of  structure  and  content  in  FLTC 
roadmaps,  and  disconnects  in  funding  control  between  the  Technology  Directorates  and 
FLTC  managers  [Mooney,  2008].  The  IRT’s  final  report  executive  summary  from 
August  2008  stated, 

“the  FLTC  process  was  making  some  progress  in  tearing  down  . . .  directorate 
stovepipes.  Several  testimonies  from  FLTC  Team  Leads  illustrated  how  new 
relationships  were  formed  among  directorates  only  as  a  result  of  the  FLTC 
construct.  However,  the  IRT  found  that  cross-directorate  focus  of  the  FLTCs  was 
reduced  by  organizational  structure  challenges”  [IRT,  2008:  3]. 

This  assessment,  along  with  the  lack  of  clear  definitions  of  each  FLTC  integration  level, 

led  the  thesis  team  to  consider  the  intent  of  the  FLTCs,  rather  than  the  specific  FLTC 

structure  as  a  way  to  represent  the  desired  integration  level  of  a  project  for  tailoring 

purposes. 

AFRL  also  seeks  to  improve  SE  application  across  all  of  its  directorates  under  the 
guidance  of  the  Systems  Engineering  Council  (SEC),  which  is  comprised  of  senior 
engineers  from  every  AFRL  Technology  Directorate.  At  the  12  August  2008  SEC  Face 
to  Face  meeting,  the  head  of  the  Council  stated  that  the  SEC’s  job  was  to  tailor  SE  and 
articulate  what  that  tailoring  means  in  order  to  affect  a  culture  shift  at  AFRL.  He  also 
said  that  “SE  is  not  just  the  things  you  do  at  the  beginning  of  a  program  to  make  your  job 
easier  ...  it’s  a  mindset”  [SEC  Meeting,  2008],  These  comments  reinforced  the  need  for 
a  tailoring  framework  to  make  SE  activities  more  accessible  to  research  scientists  and 
engineers.  The  SEC  also  provided  guidance  on  the  types  of  discriminators  to  be  used  in 
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the  project  taxonomy,  the  approach  to  gathering  case  study  candidates,  and  a 
recommendation  to  perform  some  validation  on  the  final  framework. 

Discussions  with  AFRL/XP’s  policy  staff  provided  additional  insight  on  AFRL’s 
strategic  objectives,  project  structures,  and  SE  policies.  AFRL  currently  places  emphasis 
on  three  Core  Processes  to  differentiate  management  practices  between  long-term 
research  (CP-1),  Program  Office  transition  projects  (CP-2),  and  projects  intended  to 
transition  urgent  warfighter  needs  directly  to  operations  (CP-3)  [ERP  CP2,  2008;  ERP 
CP3,  2008].  These  discussions  also  further  explained  how  AFRL  implements  FLTCs  as 
a  project  management  and  integration  structure.  Concern  was  raised  over  some 
technology  managers  erroneously  reporting  Technology  Readiness  Level  (TRL)  status,  so 
a  recommendation  was  made  to  not  use  it  as  a  project  discriminant  for  this  project; 
however,  the  team  found  TRLs  to  be  the  best  measure  of  technology  maturity  on  S&T 
projects  and  chose  to  use  them  as  a  discriminant  in  the  SE  Framework.  Finally,  the  XP 
staff  provided  several  recommendations  for  good  case  study  projects  to  evaluate  in  this 
research  project  [XP  Meeting,  2008]. 

Recognizing  the  need  to  identify  the  criteria  for  transitioning  a  product,  “The  Manager’s 
Guide  to  Technology  Transition  in  an  Evolutionary  Acquisition  Environment,”  was 
released  in  January  2003.  Transitioning  refers  to  a  product  being  usable,  producible, 
reliable,  and  affordable  [Guide  to  TT,  2003].  The  Guide  identifies  the  usability  criteria  as 
nine  distinct  Technology  Readiness  Levels  to  assess  technology  maturity.  The  remaining 
criteria  (producible,  reliable,  and  affordable)  are  identified  as  five  distinct  Engineering 
and  Manufacturing  Readiness  Levels  (EMRLs  or  MRLs)  (Table  1). 
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Table  1:  TRL  and  EMRL  Definitions  and  Mapping 


Environment",  Version  1 .0,  January 31 , 2003,  Pg  2-22 


The  level  of  S&T  project  concept  and  technical  maturity  has  a  direct  link  to  the  budget. 
The  DoD  Financial  Management  Regulation  Volume  2B,  Chapter  5,  July  2008,  defines 
seven  budget  activities,  to  include:  Basic  Research,  Applied  Research,  Advanced 
Technology  Development  (ATD),  Advanced  Component  Development  and  Prototypes 
(ACD&P),  System  Development  and  Demonstration  (SDD),  RDT&E  Management 
Support,  and  Operational  System  Development  [Finance  Management,  2008].  These 
activities  serve  as  the  basic  structure  for  the  various  types  of  development  project  funding 
and  are  strictly  controlled  and  monitored.  The  DoD  research  community  (including 
AFRL)  most  commonly  uses  the  6.1  (basic  research),  6.2  (applied  research),  and  6.3 
(ATD)  budget  activity  codes  for  funding  S&T  projects. 

Obstacles  to  SE  Success 

While  the  recommendations,  guidance  and  policies  for  increased  SE  and  integration  are 
in  place,  SE  has  not  yet  flourished  within  AFRL’s  working  levels.  Some  project 
managers  within  AFRL  resist  this  initiative,  claiming  that  SE  activities  were  developed 
for  major  acquisition  programs  and  will  “stifle  the  creativity”  required  for  S&T  projects. 
Others  decry  the  “burden”  on  time  and  fiscal  resources  of  implementing  a  comprehensive 
SE  program  for  relatively  “small”  laboratory  efforts.  There  exists  a  perception  of 
projects  being  constrained  by  bureaucratic  boundaries,  whether  organizational,  funding 
type,  or  transition  path.  For  project  leaders  in  a  “technology  push”  paradigm,  performing 
systems  engineering  with  only  “soft  requirements  and  changing  customers”  can  appear  to 
be  a  waste  of  time  and  money  [TASE,  2006:  29]. 
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A  primary  obstacle  to  proper  implementation  of  SE  within  AFRL  is  cultural  [Coglitore  et 
al,  2004:  25].  The  other  major  impediment  to  AFRL’s  comprehensive  adoption  of  SE  is 
the  lack  of  “formal,  specialized  tools  supporting  systems  engineering  sub-disciplines” 
[TASE,  2006:  29].  AFRL’s  Materials  and  Manufacturing  Directorate  (AFRL/RX) 
recognized  the  tie  between  culture  and  tools  and  requested  AFIT  investigate  how  to 
encourage  the  use  of  SE  processes,  methods  and  tools  within  the  directorate.  The 
research  identified  that  one  of  the  cultural  impediments  to  embracing  SE  was  the 
overwhelming  amount  of  recommended  activity  in  typical  SE  documentation.  After 
consulting  with  a  senior  systems  engineer  from  AFRL/RX,  the  team  determined  that  a 
tool  or  framework  could  be  developed  to  tailor  the  large  amount  of  generic  SE  practices 
to  specific  S&T  projects  at  various  levels  of  size  and  maturity,  mitigating  some  of  the 
cultural  arguments  against  SE.  Additionally,  a  tool  could  simplify  the  complex  SE 
universe  for  those  who  desire  to  use  SE  but  don’t  know  where  to  start  for  their  project, 
allowing  SE  implementation  more  pervasively  within  the  labs.  Based  on  direction  from 
the  SE  Council,  the  delivered  SE  tailoring  framework  is  intended  for  ubiquitous  use  by 
all  of  AFRL,  not  just  one  directorate.  [SEC  Meeting,  2008]. 

SE  Policies/Directives  and  Best  Practices 

To  provide  a  comprehensive  SE  framework,  the  research  team  needed  to  clearly 
understand  the  breadth  and  depth  of  SE  activities.  Thus  began  a  detailed  literature 
search,  ranging  from  Air  Force  policies,  to  professional  SE  organization  publications,  to 
academic  textbooks. 
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At  the  Air  Force  level,  policies  related  to  SE  concepts  and  methods  display  the  Air 
Force’s  understanding  of  the  importance  of  SE.  AFI  10-604,  “Capability  Based  Planning 
(CBP),”  requires  a  process  to  be  analytically  sound,  repeatable,  and  traceable  in  order  to 
identify,  assess,  and  prioritize  capability  needs  and  potential  tradespace  study  areas 
[Capabilities  Based  Planning,  2006:  3].  AFI  63-1201,  “Acquisition,  Fife  Cycle  Systems 
Engineering”  identifies  the  SE  methods  and  management  required  to  provide  and  sustain 
products/systems,  to  be  cost-effective,  operationally  safe,  and  effective  [Fife  Cycle  SE, 
2007:  1].  AFI  63-101,  “Operations  of  Capabilities  Based  Acquisition  System,”  is  a  guide 
to  for  a  systematic  framework  approach  when  acquiring  AF  capabilities  [Capabilities 
Based  Acquisition,  2005:  1].  The  Air  Force  also  communicates  best  SE  practices  in 
manners  other  than  policies.  The  SE  Assessment  Model  (SEAM)  describes  a  set  of  SE 
best  practices  tailored  for  use  by  Air  Force  programs  and  projects.  The  model  facilitates 
self  assessment  and  independent  assessment  of  SE  implementation  on  individual  projects 
[SEAM,  2008], 

Air  Force  Materiel  Command  (AFMC)  documents  further  refine  SE  concepts  and 
methods.  AFMCI  61-102,  “Advanced  Technology  Demonstration  Technology  Transition 
Planning”  provides  an  outline  of  policy  and  organizational  responsibilities  for  managing 
and  transitioning  Advanced  Technology  Demonstrations  (ATDs)  [ATD  Planning,  2006: 
1].  Additionally,  the  Technology  Program  Management  Model  (TPMM)  provides  a 
logical  methodology  to  plan  and  develop  programs  via  stage  gates.  TPMM  is  currently 
being  implemented  within  AFMC  and  AFRL  as  the  Developing  &  Sustaining  Weapons 
Systems  (D&SWS)  initiative  [Technology  Transitions,  2008]. 
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Non-policy  documents  utilized  within  the  SE  community  include  the  DoD  Defense 
Acquisition  Guidebook  (DAG),  the  Friedman-Sage  Matrix,  and  the  INCOSE  Handbook. 
The  DAG  delineates  16  SE  processes.  The  eight  Technical  Processes  (TPs)  include 
Requirements  Development,  Logical  Analysis,  Design  Solution,  Implementation, 
Integration,  Verification,  Validation,  and  Transition.  The  eight  Technical  Management 
Processes  (TMPs)  include  Decision  Analysis,  Technical  Planning,  Technical  Assessment, 
Requirements  Management,  Risk  Management,  Configuration  Management,  Technical 
Data  Management,  and  Interface  Management  (Figure  1)  [DAG,  2004],  The  AF  Center 
for  Systems  Engineering  Case  Studies  include  the  Friedman-Sage  Matrix,  which 
illustrates  nine  key  SE  concept  areas,  representing  phases  in  the  systems  engineering 
lifecycle  and  necessary  process  and  systems  management  support  [Friedman-Sage, 

2005].  The  International  Council  on  Systems  Engineering  (INCOSE),  a  leading  SE 
professional  organization,  published  the  INCOSE  Systems  Engineering  Handbook  in 
June  2006.  This  handbook  provides  key  SE  process  activities  at  a  detailed  level,  with  the 
purpose  of  designing  for  affordability  and  performance.  The  handbook  tends  to  focus  on 
industry-related  projects,  in  an  input,  control,  output,  mechanism  (ICOM)  format 
[INCOSE,  2006], 

The  2008  AFRL  Technology  Maturity  Conference  was  another  information  resource  for 
the  practice  of  SE  in  defense-related  industry.  A  common  theme  at  the  conference  was 
the  use  of  various  readiness  levels  to  determine  the  ability  of  a  product  to  transition.  The 
conference  provided  awareness  to  the  team  of  current  practices  to  mature  and  transition 
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Figure  1:  Defense  Acquisition  Guidebook  Processes 


“grown  up”  technologies  to  program  offices.  It  also  solidified  the  need  for  a  tailored 
approach  rather  than  defining  a  new  discriminant  by  which  to  measure  projects. 

In  addition  to  the  contributions  by  the  SE  community  as  stated  above,  the  academic 
community  is  also  a  significant  resource.  “The  Engineering  Design  of  Systems:  Models 
and  Methods”  by  Dennis  Buede  is  a  text  utilized  at  the  graduate  level,  and  addresses 
methods  for  using  models  during  the  SE  process  [Buede,  1999].  This  text  is  one  of  the 
two  resources  for  the  framework  that  provides  SE  methods  at  a  detailed  level;  the  other 
being  the  INCOSE  Handbook.  The  team  examined  other  academic  contributions  during 
the  literature  review,  but  settled  on  the  Buede  text  as  our  primary  reference.  “Essentials 
of  Project  and  Systems  Engineering  Management”  by  Howard  Eisner,  also  a  text  utilized 
at  the  graduate  level,  provides  an  organization  of  30  SE  elements,  which  span  the  overall 
SE  process  over  a  system’s  life  cycle  [Eisner,  2002].  “Best  Project  Management  and 
Systems  Engineering  Practices  in  the  Pre-acquisition  Phase  for  Federal  Intelligence  and 
Defense  Agencies”  by  Steven  R  Meier,  was  published  in  Project  Management  Journal,  in 
March  2008.  Meier  concludes  that  SE  must  be  upfront  and  include  an  understanding  of 
the  interfaces,  technology  assessments,  system  trades,  and  risk  management  [Meier, 
2008].  These  documents  helped  the  research  team  establish  a  SE  knowledge  base  to 
proceed  with  building  the  tailoring  framework. 

Differences  from  Previous  Efforts 

Earlier  activities  looked  at  the  topic  of  early  application  of  SE  in  technology  development 
and  acquisition.  While  these  studies  addressed  many  of  the  same  SE  processes  and 
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methods  as  this  thesis,  the  scope  of  applicable  projects,  intended  implementation  of  the 
final  results,  and  specific  deliverables  are  different.  The  reports  and  presentations  should 
all  be  considered  when  mapping  out  an  SE  program,  but  this  thesis  project  does  in  fact 
stand  alone  as  a  comprehensive  guide  and  tool  to  tailoring  SE  activities  for  S&T  projects. 

One  of  the  primary  studies  that  relates  to  this  thesis  topic  area  is  the  TASE  report  [TASE, 
2006].  The  TASE  report  focused  on  documenting  the  state  of  SE  implementation  within 
AFRL  and  recommending  ways  to  improve  its  overall  use  on  S&T  projects.  The  report 
looked  at  consistent  application  of  all  SE  processes  to  all  projects,  and  while  general 
tailoring  was  recommended,  a  specific  tailoring  framework  was  out  of  scope.  This  thesis 
delivers  a  specific  model  for  tailoring  SE  activities  to  a  range  of  S&T  projects  based  on 
maturity,  size,  Core  Process  category,  and  funding  source.  The  TASE  report  also  focused 
primarily  on  Advanced  Technology  Demonstrations  (ATDs),  which  are  only  a  subset  of 
the  type  of  projects  contained  within  AFRL’s  portfolio.  This  research  looks  at  the  entire 
range  of  AFRL  projects.  TASE  used  the  16  SE  processes  from  the  DAG,  which  is 
consistent  with  SE  taxonomy  approach  used  in  this  thesis.  Both  the  TASE  report  and  this 
thesis  include  a  comprehensive  review  of  existing  AFRL  SE  policy  and  guidance,  but  the 
TASE  report  additionally  focused  on  the  cultural  effects  of  implementing  SE  within 
AFRL  -  something  this  thesis  does  not  specifically  address.  There  are  also  many 
similarities  in  the  methodologies  between  this  thesis  and  the  TASE  report:  literature 
review  (including  policies,  guidance,  and  non-DoD  SE  practices),  assessment  of  past  and 
current  AFRL  projects,  and  recommendations  of  applicable  SE  processes,  methods  and 
tools.  The  research  additionally  reviews  a  non-DoD  case  study  for  an  outside  perspective 
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on  research  and  development  activities.  Finally,  the  deliverables  between  the  two  efforts 
differ,  as  TASE  produced  two  reports  (assessment  and  recommendations),  while  the 
research  will  deliver  a  single  report  along  with  an  SE  tailoring  tool  that  can  be 
immediately  used  by  S&T  project  leaders  to  identify  a  recommended  level  of  SE  rigor  for 
their  specific  project  [TASE,  2006]. 

Another  applicable  study  is  the  Commission  on  Pre-Milestone  A  (Pre-MS  A)  Systems 
Engineering  report  [Pre  MS-A,  2008].  The  Pre-MS  A  report  addressed  the  effects  of 
early  implementation  of  SE  on  major  acquisition  programs  but  did  not  specifically 
address  S&T  projects.  It  placed  emphasis  on  the  Concept  Refinement  and  Technology 
Development  phases  of  the  systems  acquisition  life  cycle  and  defined  a  minimum  level  of 
early-phase  SE  activities  for  programs  that  follow  this  model.  The  report  described 
“general  policies  and  best  practices  for  systems  engineering  in  all  phases,”  but  while 
many  of  the  policies  and  practices  that  the  Pre-MS  A  report  recommended  are  also 
applicable  for  S&T  projects  (which  are  usually  smaller  in  size),  it  was  not  focused 
specifically  on  the  AFRL  project  portfolio  [Pre  MS-A,  2008:  72].  The  Pre-MS  A  report 
generically  recommended  tailoring,  saying  “Formal  SE  processes  should  be  tailored  to 
the  application”,  but  no  specific  tailoring  recommendations  were  made  [Pre  MS-A,  2008: 
7].  The  Pre-MS  A  report  also  contained  a  thorough  review  of  the  training  and  experience 
of  the  Air  Force’s  acquisition  workforce,  which  is  well  outside  of  the  scope  of  this  thesis, 
but  well  within  the  realm  of  actions  necessary  within  AFRL.  Again,  there  were 
similarities  in  the  Pre-MS  A  and  AFIT  methodologies,  notably  a  review  of  previous  SE 
reports  and  an  emphasis  on  case  studies  to  produce  a  report  and  recommendations.  This 
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thesis,  however,  looks  at  S&T  and  AFRL-specific  policy  and  guidance  in  developing  the 
analysis  approach,  where  the  Pre-MS  A  report  looked  primarily  at  prior  review  panel 
reports  for  its  approach.  The  Pre-MS  A  Commission  delivered  a  report  that  was  centered 
on  “trying  to  define  a  minimum  set  of  systems  engineering  processes”  as  well  as  a  list  of 
20  questions  that  should  be  asked  on  all  programs  prior  to  Milestone  A  [Pre  MS- A,  2008: 
1,3].  The  thesis  deliverable  covers  many  of  these  SE  minimums  and  questions,  but 
includes  more  SE  detail  and  allows  tailoring  of  those  details  based  on  the  type  of  S&T 
project  being  considered. 

During  the  course  of  this  research  project,  AFRL/RX  developed  a  streamlined  approach 
using  IPPD  and  AFRLI  61-104  for  applying  SE  principles  to  their  programs.  The 
approach  recommends  AFRL/RX  tailoring  to  the  eight  key  questions,  showing  the 
amount  of  effort  recommended  for  four  project  types  (basic  research,  applied  research, 
advanced  research,  and  Advanced  Technology  Demonstrations)  [Malas,  2008].  This 
streamlined  SE  approach,  intended  as  a  bottoms-up  minimum  set  of  activities,  omitted 
many  general  and  detailed  SE  activities.  Although  the  streamlined  approach  and  the 
thesis  framework’s  purposes  are  similar  in  nature,  the  framework  presents  a  much  more 
detailed  and  comprehensive  top-down  approach,  providing  tailoring  of  a  greater  range  of 
SE  activities  for  the  complete  set  of  potential  AFRL  project  states. 

Purpose  for  Research 

While  this  research  clearly  builds  on  the  work  done  by  previous  studies,  especially  the 
TASE  report,  it  stands  on  its  own  as  a  comprehensive  review  of  all  SE  processes. 
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methods,  and  tools  as  they  apply  to  the  wide  range  of  S&T  project  types.  Most 
importantly,  it  provides  a  useful  tool  for  customizing  the  amount  of  rigor  put  into  each  of 
these  SE  principles.  The  intent  of  this  deliverable  is  to  not  only  make  the  case  for 
increased  focus  on  SE  within  AFRL,  but  to  facilitate  the  implementation  of  select  SE 
activities  at  a  level  appropriate  for  specific  S&T  projects. 
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III.  Methodology 


Methodology  Overview 

This  chapter  addresses  the  development  and  validation  of  the  SE  tailoring  framework, 
comprised  of  a  taxonomy  of  comprehensive  SE  activities  and  a  separate  taxonomy  of 
relevant  categories  and  domain  values  possible  for  S&T  projects.  This  framework  forms 
the  basis  of  the  tailoring  tool  discussed  in  Chapter  IV.  The  SE  taxonomy  incorporates  a 
broad  set  of  recognized  activities  from  academic,  defense,  and  industry  sources  and 
organizes  these  activities  according  to  the  Defense  Acquisition  Guidebook’s  structure  of 
Technical  Processes  (TPs)  and  Technical  Management  Processes  (TMPs).  The  project 
taxonomy  forms  the  basis  for  treating  each  project  as  a  state  problem,  with  unique  project 
discriminants  consisting  of  discrete  domain  values.  Both  taxonomies  were  developed 
independently  and  then  matrixed  into  the  SE  framework.  These  groupings  allow  for 
specific  tailoring  as  a  function  of  unique  project  characteristics.  The  framework 
validation  was  accomplished  by  analyzing  current  and  recently  completed  S&T  projects 
as  well  as  review  by  prominent  systems  engineers  within  AFRL. 

SE  Taxonomy  Development 

As  described  in  Chapter  II,  a  variety  of  approaches  exist  for  implementing  Systems 
Engineering  in  developmental  and  S&T  projects;  however,  the  team  did  not  find  a  single 
literature  source  that  included  a  sufficiently  comprehensive  and  appropriate  set  of  these 
activities  for  direct  transfer  into  the  desired  SE  taxonomy.  The  team  determined  that 
pulling  from  multiple  literature  sources  would  allow  for  a  look  at  systems  engineering 
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from  academic,  defense,  and  industry  perspectives  and  would  provide  the  basis  for 
generating  a  “superset”  of  activities  that  spanned  all  three  realms  of  experience. 

Academic  textbooks  often  address  the  SE  process  as  a  whole,  but  focus  instruction  on  the 
author’s  specific  areas  of  interest  and  communicate  in  the  author’s  preferred  terminology. 
Defense  sources  outline  policy  directives  regarding  SE  activities  for  acquisition  projects, 
but  do  not  specifically  state  which  activities  are  appropriate  for  various  project  types. 
Industry  sources  encompass  accepted  practices  from  a  wide  variety  of  business  sectors 
and  introduce  a  level  of  specificity  not  found  in  either  academic  or  defense  sources.  In 
fact,  no  readily-available  sources  were  found  to  address  appropriate  SE  activities 
specifically  for  S&T  projects.  As  all  three  literature  categories  showed  promise  for 
contribution  to  the  “superset”,  the  group  selected  a  single  source  from  each  category  to 
incorporate  into  the  SE  taxonomy.  The  selected  sources  from  the  literature  review  are 
“The  Engineering  Design  of  Systems:  Models  and  Methods”  by  Dennis  Buede  [Buede, 
1999],  Chapter  4  from  the  Defense  Acquisition  Guidebook  (DAG)  [DAG,  2004],  and  the 
International  Council  for  Systems  Engineering  (INCOSE)  Handbook  [INCOSE,  2006]. 

With  sources  identified,  the  research  analyzed  candidate  organization  schemes  to 
determine  the  most  appropriate  construct  for  the  SE  taxonomy.  These  organization 
schemes  included  the  Friedman-Sage  Matrix  [Friedman-Sage,  2005]  used  on  multiple  Air 
Force  Center  for  Systems  Engineering  case  studies,  The  Thirty  Elements  of  Systems 
Engineering  from  Chapter  7  of  the  Eisner  textbook  referenced  in  Chapter  II  [Eisner, 

2002:  191,  194],  and  the  DAG  TPs  and  TMPs  [DAG,  2004].  As  the  direct  application  of 
the  research  effort  is  defense  S&T,  and  to  remain  consistent  with  current  AFRL  policies 
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and  practices,  the  team  determined  that  the  DAG  construct  was  most  appropriate.  An 
added  benefit  of  the  DAG  construct  is  the  inclusion  of  the  systems  engineering  “Vee”,  an 
iterative  approach  to  implementing  TPs  on  a  project,  as  well  as  the  continuous 
implementation  of  TMPs  on  a  project.  The  importance  of  using  the  SE  “Vee”  was  noted 
by  the  AFRL  FLTC  Independent  Review  Team  (IRT):  “The  IRT  used  the  SE  “Vee” 
from  the  Defense  Acquisition  Guidebook  (DAG)  as  interpreted  for  tailored  application  to 
the  science  and  technology  environment  in  AFRFI  61-104  as  the  basis  for  evaluating 
FFTC  SE  processes”  [IRT,  2008],  Additionally,  AFRL  currently  bases  their  SE  planning 
activities  (AFRLI  61-104)  around  the  DAG  processes,  providing  a  familiar  taxonomy  for 
users  of  the  Tailored  SE  Framework  (Figure  2)  [S&T  SE,  2008], 

The  team  gathered  systems  engineering  processes  from  each  of  the  3  literature  references 
and  compiled  them  into  16  lists  according  to  the  8  TPs  and  8  TMPs  from  the  DAG.  Once 
the  data  was  gathered,  each  list  was  organized  by  functional  hierarchy  and  chronological 
order,  resulting  in  a  completed  “superset”  of  systems  engineering  processes  in  each  of  the 
16  categories.  Within  each  category,  the  SE  activities  were  organized  according  to 
functional  groupings,  in  chronological  order  (to  the  extent  possible),  and  hierarchically 
according  to  five  levels  of  increasing  detail.  The  DAG  processes  comprise  Level  1  of  the 
SE  taxonomy,  functional  groupings  make  up  Level  2,  and  most  of  the  executable 
activities  reside  at  Level  3.  Levels  4  and  5  contain  details  or  variations  of  the  Level  3 
activities  that  can  be  selectively  applied  at  the  discretion  of  the  project  (Table  2).  With 
the  elements  of  the  SE  taxonomy  established,  the  SE  processes  were  organized  into  a 
single  Excel  worksheet  to  provide  optimum  visibility  and  management  of  the  resulting 
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Figure  2:  Mapping  of  AFRLI 61-104  Questions  to  Defense  Acquisition  Guidebook  Processes 


hierarchy,  function  and  structure  of  the  “superset”  within  the  tool  to  be  discussed  in 
Chapter  IV. 


Table  2:  SE  Taxonomy  Levels  and  Activities 


SE  Taxonomy  Hierarchy 

Description 

#  of  SE  Activities 

Level  1 

16  DAG  Processes  + 
Fundamental  Principles 

17 

Level  2 

Functional  SE  Activity 
Grouping 

65 

Level  3 

Tailored  SE  Activities 

350 

Level  4 

Detailed  SE  Activities 

538 

Level  5 

Detailed  SE  Activities 

161 

During  review  of  the  16  DAG  processes,  six  common  SE  activities  were  discovered  in 
multiple  processes.  These  activities  were  all  from  the  INCOSE  Handbook  and  related  to 
utilization  of  existing  processes  and  practices  within  a  larger  enterprise  management 
structure.  Examples  of  these  common  activities  are  “utilize  enterprise  strategic  plan”  and 
“utilize  enterprise  infrastructure”.  Rather  than  leave  these  redundant  activities  buried 
within  multiple  categories,  an  additional  category  titled  “Fundamental  Principles”  was 
created  at  Level  1,  with  a  roll-up  at  Level  2  titled  “Utilize  Enterprise  Capabilities”. 


Project  Taxonomy  Development 

Similar  to  the  SE  framework,  the  research  assessed  multiple  organization  schemes  to 
encompass  the  characteristics  used  by  AFRL  to  discriminate  between  projects.  Initial 
organization  attempts  to  establish  relationships  between  the  individual  project 
discriminants  resulted  in  a  layered  matrix  schema  with  project  size  and  complexity  along 
the  vertical  axis,  project  maturity  along  the  horizontal  axis,  and  the  DAG’s  16  processes 
forming  the  depth  of  the  matrix.  While  this  initial  attempt  provided  an  understanding  of 
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the  relationships  between  the  discriminants,  it  was  not  sufficient  to  capture  the  possible 
combinations  of  domain  values  within  the  discriminants  in  enough  detail  to  adequately 
tailor  the  SE  activities  for  a  particular  project.  Discussions  with  thesis  advisors  regarding 
the  multiple  discriminants  implemented  by  AFRL  resulted  in  a  decision  to  treat  the 
project  taxonomy  as  a  state  problem,  with  the  project  state  being  determined  by  the 
applicable  domain  values  within  each  discriminant  for  the  project.  In  essence,  the  desired 
tool  should  provide  a  transformation  of  the  project  state  description  into  a  set  of  systems 
engineering  activities  with  recommended  amounts  of  rigor  to  be  applied  to  the  project. 


The  project  taxonomy  initially  included  all  established  discriminants  currently  used  by 
AFRL  (Figure  3).  The  team  conducted  meetings  in  August  2008  with  AFRL/XP  and 
with  the  AFRL  Systems  Engineering  Council  to  refine  the  potential  list  of  discriminants 


Potential  “DISCRIMINANTS” 

aa,&i&i)G5  »®K«a 


■  Primary  Funding  (6.1 , 6.2,  6.3,  other) 

■  Secondary  Funding  (6.1,  6.2,  6.3,  other) 

■  Funding  Amount  (<$200K,  $200K  -  $2M,  $2M  -  $20M,  >$20M) 

■  Core  Process  (CP-1 ,  CP-2,  CP-3,  other) 

«  Technology  Readiness  Level  (TRL)  (1  -  9) 

■  Manufacturing  Readiness  Level  (MRL)  (1-10) 

«  FLTC  Level  (FLTC,  Problem,  Attribute,  Product,  Program) 

■  Management  Level  (Multi-Dir,  Dir,  Div,  Branch,  PM) 

■  Strategic  Goals  (not  currently  planned  for  study) 

■  Requirements  Maturity  (Tech  Push,  Rqmts  Pull) 


Results  in  460,800+  possible  STATES  -  unwieldy  for  everyone 


Integrity  -  Service  -  Excellence  6 


Figure  3:  Potential  Project  Discriminants  Presented  to  AFRL  SE  Council  (Aug  ’08) 
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for  the  final  project  taxonomy  and  made  further  refinements  during  the  taxonomy 
development  process  to  ensure  the  final  discriminants  were  reduced  to  a  manageable  set, 
were  appropriate  for  S&T  projects  and  contained  discrete  domain  values  for  each 
discriminant.  The  SE  Council  recommended  eliminating  the  “Secondary  Funding”, 
“Manufacturing  Readiness  Level  (MRL)”,  “Management  Level”,  and  “Strategic  Goals” 
discriminants,  as  well  as  the  “other”  domain  value  for  the  “Core  Process”  discriminant 
[SEC  Meeting,  2008], 

First,  the  “Secondary  Funding”  discriminant  was  initially  included  to  account  for  multiple 
funding  sources  for  a  project,  but  was  eliminated  with  guidance  that  a  project  should  be 
tailored  according  to  the  “highest”  level  of  funding.  An  example  would  be  treating  a 
project  with  funding  from  both  6.2  and  6.3  categories  as  a  6.3  project.  Second,  as  the 
domain  values  (1-10)  within  the  “Manufacturing  Readiness  Level  (MRL)”  discriminant 
have  direct  correlation  with  the  domain  values  (1-9)  of  the  “Technology  Readiness  Level 
(TRL)”  discriminant,  it  was  also  eliminated.  Next,  the  “Management  Level”  and 
“Strategic  Goals”  discriminants  were  initially  included  to  account  for  projects  spanning 
multiple  directorates  within  AFRL,  but  were  eliminated  in  favor  of  the  “Focused  Long 
Term  Challenge  (FLTC)”  discriminant,  which  also  incorporates  dependencies  between 
AFRL  directorates  associated  with  a  particular  project.  Finally,  the  team  eliminated  the 
“other”  domain  value  for  the  “Core  Process”  discriminant  upon  discovery  that  “CP-1”, 
CP-2”,  and  “CP-3”  encompassed  the  entire  domain. 
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With  a  refined  set  of  discriminants  in  hand,  the  research  defined  each  of  the  discriminants 


and  associated  domain  values  in  the  form  of  a  questionnaire  to  be  provided  to  projects 
during  the  Case  Study  phase  of  the  thesis.  Research  into  the  definitions  aided  in  the 
team’s  understanding  of  each  discriminant  and  resulted  in  further  refinements  to  the 
project  taxonomy.  The  first  refinement,  eliminating  the  “other”  domain  value  from  the 
“Primary  Funding”  discriminant,  resulted  from  further  discussions  with  AFRL/XP 
regarding  the  expected  funding  sources  for  S&T  projects.  The  second  refinement 
modified  the  “FLTC”  discriminant,  partly  due  to  a  lack  of  a  formal  definition  of  the 
discrete  qualifiers  between  the  proposed  domain  values  (“Challenge”,  “Problem”, 
“Capability  Concept”,  “Product”  and  “Program”),  but  more  particularly  to  implement  a 
more  generic  description  of  a  project’s  “Integration  Level”  with  the  discrete  domain 
values  of  “Subsystem  Level  Technology”,  “System  Level  Concept”,  and  “Mission  Level 
Concept”.  This  change  also  makes  the  framework  more  accessible  to  S&T  projects 
outside  of  AFRL.  The  last  refinement  to  the  project  taxonomy  consisted  of  a  slight 
adjustment  to  the  domain  values  of  the  “Funding  Amount”  discriminant,  which  occurred 
after  the  team  spent  considerable  time  “tailoring”  the  SE  framework,  and  better 
encapsulates  discrete  funding  breaks  at  which  certain  SE  activities  become  appropriate 
for  projects.  The  discriminant  was  renamed  “Project  Budget”  with  domain  values  of 
“Less  than  $500K”,  “$500K  to  $2M”,  and  “Greater  than  $2M”. 

The  final  project  questionnaire,  provided  as  Appendix  A  to  this  thesis,  provides 
definitions  for  each  of  the  final  6  discriminants  and  18  domain  values  (Figure  4)  and 
seeks  to  define  the  particular  “state”  of  the  project  to  which  the  SE  tailoring  will  be 
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applied.  The  final  discriminants  and  domain  values  result  in  over  648  possible  project 
states,  a  significant  reduction  from  the  original  discriminant  set,  which  contained  over 
460,000  possible  states.  The  need  to  address  each  of  these  potential  states  in  the  project 
taxonomy  mandated  that  the  delivered  tool  provide  a  simple  interface  with  the  flexibility 
to  report  results  for  any  number  of  the  project  discriminants  and  domain  values  (which 
potentially  increases  the  number  of  states  above  648). 


Project  Taxonomy 


•  Define  “STATE”  of  an  S&T  Project  by  its  “DISCRIMINANTS” 

-  “STATE”  =  Intersection  of  “DISCRIMINANTS” 

•  DISCRIMINANT  #1  (DOMAIN  VALUE) 

■  DISCRIMINANT  #2 

■ 

■  DISCRIMINANT  #x: 

■  Final  Discriminants  (Domain  Values) 

■  RDT&E  Category  (6.1 , 6.2,  6.3) 

■  Project  Budget  (<  $500K,  $500K  <  $  <  $2M,  >  $2M) 

■  Core  Process  (CP-1 ,  CP-2,  CP-3) 

■  Technology  Readiness  Level  (1-2,  3-4,  5-6,  7-9) 

■  Integration  Level  (Subsystem,  System,  Mission) 

■  Requirements  Maturity  (Technology  Push,  Requirements  Pull) 


Figure  4:  Discriminants  and  Domain  Values  (Project  Taxonomy) 

Systems  Engineering  “Tailoring” 

The  goal  of  the  tailoring  effort  was  to  indicate  the  relative  importance  for  each  SE 
activity  for  a  given  project,  based  on  the  project’s  specific  domain  values.  Returning  to 
the  previous  state  analogy,  the  tailoring  process  is  the  mechanism  to  transform  the  project 
state  description  to  the  recommended  set  of  activities  with  associated  rigor  levels.  The 
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tailoring  effort  and  implementation  into  the  Excel  tool,  “Systems  Engineering  Tailoring 
Tool  for  Science  &  Technology  Projects  (SETT-STP)”  followed  a  four  step  process:  1) 
Tailor  at  Level  3  of  the  SE  Taxonomy,  2)  Implement  Discriminant  Tailoring  into  the  SE 
Framework,  3)  Normalize  SE  Rigor  Values  (0-100%  Scale),  and  4)  Apply  Tailoring 
Factors  to  Gauge  Impact  of  Various  Schemes  to  SE  Rigor. 

First,  the  research  looked  at  350+  Level  3  SE  activities,  methods  and  tools  listed  in  the 
SE  taxonomy  (Appendix  F),  and  determined  the  applicability  of  each  activity  to  the  18 
project  domain  values.  The  research  team  created  a  table  of  SE  activities  for  each  TP  and 
TMP,  as  well  as  for  each  discriminant  category  and  domain  value.  Each  SE  activity  at 
Level  3  of  the  SE  taxonomy  was  evaluated  independently  against  each  domain  value. 

For  instance,  if  the  research  explored  applicability  of  an  activity  for  the  6.2  RDT&E 
Budget  domain  value,  no  assumptions  were  made  as  to  the  associated  TRL,  Core  Process, 
or  Integration  Level.  As  a  general  guideline,  the  tailoring  was  more  inclusive  for  larger, 
more  expensive,  and  more  mature  S&T  projects.  Likewise,  the  evaluation  tended  to 
tailor  out  more  activities  for  smaller,  less  expensive  and  immature  projects.  This 
tendency  was  utilized  several  times  when  there  was  debate  over  whether  an  activity  was 
applicable  or  not  for  a  given  domain  value.  Specifically,  a  6.3,  $5  million,  and  TRL-8 
project  was  found  to  require  more  SE  rigor  than  a  6.1,  $250  thousand,  and  TRL-3  project; 
the  framework  needed  to  reflect  this  in  a  quantitative  manner.  A  sample  tailoring  table 
from  TP-4  “Implementation”  is  shown  in  Table  3.  At  least  two  students  reviewed  each 
set  of  tables  for  each  TP  and  TMP  to  ensure  consistency.  After  several  internal  review 
and  discussion  sessions,  the  tables  were  submitted  to  the  faculty  advisor  for  review  and 
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Table  3:  Sample  SE  Tailoring  Table  for  TP-4  “Implementation' 
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comment.  This  review  process  resulted  in  changes  to  not  only  the  tailoring,  but  also  to 
the  grouping  and  wording  of  the  activities.  A  couple  of  noteworthy  trends  emerged  as  the 
tailoring  effort  progressed.  Two  of  the  discriminants.  Integration  Level  and 
Requirements  Maturity,  did  not  lend  themselves  to  significant  tailoring  between  the 
respective  domain  values.  For  Integration  Level,  the  research  found  that  the  same  SE 
activities  applied  whether  the  product  under  development  was  a  subsystem,  a  self- 
contained  system,  or  part  of  a  mission-level  concept.  The  Requirements  Maturity 
discriminant  resulted  in  significant  tailoring  in  TMP-4  (Requirements  Management),  but 
was  largely  consistent  between  the  Technology  Push  and  Requirements  Pull  domain 
values  for  the  other  processes  in  the  SE  Taxonomy.  The  team  debated  removing  these 
discriminants,  but  decided  that  it  was  important  to  include  them  in  the  framework  to  build 
credibility  with  the  wide  range  of  projects  that  would  potentially  use  the  framework.  An 
effort  was  made,  though,  to  reduce  the  impact  of  these  two  discriminants  on  the  reported 
output  of  SETT-STP  through  the  Tailoring  Factors,  discussed  in  the  third  step  of  the 
tailoring  process. 

Another  trend  discovered  in  the  tailoring  process  resulted  in  a  change  to  the  Project 
Budget  discriminant,  with  initial  domain  values  of  “Less  than  $200K”,  “$200K-$2M”, 
“$2M-$20M”,  and  “Greater  than  $20M”.  In  the  first  tailoring  iteration,  the  only  real 
effect  of  tailoring  the  Project  Budget  discriminant  was  observed  in  the  “Less  than 
$200K”  category,  as  this  was  such  a  small  budget  level  that  not  much  formal  systems 
engineering  activity  could  be  afforded  without  undercutting  the  S&T  benefit. 
Additionally,  the  research  team  held  numerous  debates  about  the  amount  of  tailoring 


34 


appropriate  for  the  “$200K  -  $2M”  category,  as  a  $250K  project  would  apply  SE  much 
differently  than  a  $1.9M  project.  The  “$2M-$20M”  and  “Greater  than  $20M”  domain 
values  provided  no  difference  in  tailoring.  Ultimately,  the  team  decided  that  the  $200K 
threshold  was  too  low,  and  that  the  split  between  the  “$2M-$20M”  and  “Greater  than 
$20M”  categories  provided  no  value.  These  domain  value  categories  were  re-designated 
“Less  than  $500K”,  “$500K  -  $2M”,  and  “Greater  than  $2M”. 

The  second  step  of  the  tailoring  activity  involved  implementing  the  Level  3  tailoring  into 
the  Excel-based  tool.  After  the  table-based  review,  the  applicable  activities  were 
transposed  into  a  series  of  Excel  workbooks  that  scored  the  SE  activities  for  each  domain 
value.  A  binary  system  annotated  the  applicability  of  each  domain  value,  with  a  “1” 
score  indicating  that  the  SE  activity  should  be  accomplished  for  a  domain  value,  and  a 
“0”  indicating  that  the  activity  was  not  critical  for  the  domain  value.  The  binary  scoring 
system  allowed  for  a  customizable  weighting  to  be  applied  to  a  selected  set  of  project 
discriminants,  while  simplifying  the  tailoring  implementation  within  the  Excel  tool.  A 
sample  input  to  the  Excel  tool  is  shown  in  Figure  5.  The  team  did  consider  an  alternate 
scoring  system  for  the  domain  values,  where  domain  values  would  receive  a  fractional 
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Figure  5:  Sample  of  Binary  Tailoring  Input  to  SETT-STP 
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score  on  a  scale  of  zero  to  one;  however,  the  degree  of  confidence  in  the  binary  system 


was  not  quantifiable  to  the  point  that  implementing  the  fractional  value  system  provided 
any  additional  benefit. 


The  third  step  in  tailoring  the  SE  activities  required  a  summation  and  normalization  of  the 
scores  on  a  0-100%  scale.  For  a  given  set  of  discriminants,  the  tailored  weights  revealed 
the  relative  level  of  SE  rigor  that  should  be  applied  to  each  activity  for  a  specific  project 
state.  The  intent  behind  this  methodology  was  to  give  a  user  a  relative  indicator  as  to 
where  they  should  apply  resources.  An  activity  with  a  100%  weight  should  be 
accomplished  to  a  more  formal  and  detailed  level  over  an  activity  with  a  60%  weight. 

The  team  notionally  interpreted  the  tailored  percentages  for  SE  rigor  according  to  the 
descriptions  in  Table  4. 


Table  4:  Notional  Interpretations  for  Reported  SE  Rigor 


SE  Rigor 
Percentage 

Notional  Interpretation 

100% 

REQUIRED:  An  activity  should  be  accomplished  to  a  complete  and  formal  level  of 
planning,  coordination,  and  documentation 

70% 

RECOMMENDED:  An  activity  should  be  considered  for  planning,  coordination,  and 
documentation 

30% 

WATCH  LIST:  An  activity  should  be  considered  for  informal  planning, 
coordination,  and  documentation 

0% 

NOT  APPLICABLE:  An  activity  is  probably  not  necessary  to  project  success  and 
requires  little  or  no  planning,  coordination,  and  documentation 

The  fourth  step  of  the  tailoring  activity  applied  tailoring  factors  to  each  of  the  Project 
Taxonomy  discriminants  to  assess  their  impact  on  the  normalized  score  of  SE  rigor.  A 
detailed  sensitivity  analysis  explored  the  impact  of  tailoring  factors  to  reported  SE  rigor 
scores.  A  baseline  case  with  equal  weighting  of  the  project  discriminants  was  compared 
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to  two  additional  weighting  methods.  The  second  tailoring  factor  scheme  attempted  to 
capture  the  lack  of  tailoring  discovered  within  the  Integration  Level  and  Requirements 
Maturity  discriminants  during  the  initial  tailoring  efforts.  The  third  tailoring  factor 
scheme  further  explored  the  impact  of  the  Project  Budget  discriminant  on  the  reported  SE 
rigor.  The  sensitivity  analysis  results  are  contained  in  Chapter  IV,  and  the  complete 
sensitivity  analysis  is  presented  as  Appendix  C.  Ultimately,  the  sensitivity  analysis 
showed  that  the  third  tailoring  factor  scheme  provided  the  greatest  spread  in  tailoring 
scores  and  was  adopted  into  SETT-STP’s  initial  release. 

Case  Study  Results  &  Feedback 

A  critical  part  of  this  research  project  is  the  intersection  of  student-derived  tailoring  with 
real-world  S&T  projects  that  utilized  systems  engineering  principles.  Much  of  the  initial 
tailoring  was  accomplished  by  applying  the  team’s  academic  knowledge  and  prior 
individual  work  experience,  but  that  was  not  sufficient  to  definitively  claim  that  the 
tailored  SE  Framework  was  accurate  and  applicable  to  potential  users.  The  team  sought 
out  several  current  and  recently  completed  S&T  projects  to  fine  tune  the  initial  tailoring. 
The  reviewed  projects,  treated  as  case  studies,  in  some  cases  validated  the  framework,  but 
more  often  drove  important  changes  to  the  SE  taxonomy’s  terminology,  grouping,  or 
weightings.  The  team  developed  an  initial  listing  of  potential  case  study  candidates  based 
on  faculty  advisor  input  and  solicitation  of  the  AFRL  SE  Council  for  applicable  project 
candidates.  The  preliminary  case  study  target  list  consisted  of  projects  within  AFRL/RX, 
but  was  expanded  based  on  the  recommendation  from  the  SE  Council  that  the  tailorable 
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SE  framework  should  be  applicable  to  all  directorates  within  AFRL.  The  research  then 
formally  solicited  candidate  projects  from  all  of  the  SE  Council  representatives. 

Basic  systems  engineering  tenets  are  applicable  to  all  complex  projects,  whether 
applicable  to  the  defense  department  or  not,  and  whether  they  are  guided  by  DoD  policy 
and  guidance  or  not.  Accordingly,  the  team  also  decided  to  pursue  S&T  projects  from 
other  governmental  agencies  (non-DoD),  as  well  as  from  corporate  research  and 
development  engineering  organizations.  Industry  perspective  is  particularly  important,  as 
corporate  livelihoods  are  often  based  on  the  successes  of  S&T  projects  as  well  as  the 
appropriate  application  of  systems  engineering  principles  to  ensure  future  development, 
integration,  and  transition  activities  are  conducted  in  a  cost-effective  manner. 

Upon  receiving  an  S&T  project  point  of  contact,  the  research  sent  out  an  initial  Case 
Study  Pre-Survey  Questionnaire  (Appendix  A).  The  questionnaire  solicited  contact 
information  and  asked  the  point  of  contact  to  indicate  which  of  the  project  taxonomy 
discriminants  applied  to  the  candidate  project.  The  points  of  contact  received  full 
definitions  of  each  of  the  discriminants  to  ensure  consistent  understanding  of  the  project 
taxonomy.  After  receiving  the  completed  questionnaires,  the  team  requested  access  to 
relevant,  previously  assembled  case  study  project  documents.  The  research  reviewed  all 
provided  documentation  and  made  notes  against  a  project-specific  tailored  output  from 
the  SE  tool  using  the  cited  domain  values.  After  reviewing  all  documentation,  the 
research  formulated  questions  for  the  project  point  of  contact  to  resolve  all  discrepancies 
and  gaps.  The  ensuing  interviews  with  the  point  of  contacts  were  insightful  and  provided 
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clarification  of  confusing  terms  and  sequences  in  the  SE  framework,  added  background 
and  understanding  to  formal  documentation,  and  illustrated  why  certain  SE  processes, 
methods  and  tools  were  or  were  not  used  for  the  project.  After  each  documentation 
review  and  personal  interview,  the  research  revised  the  tailored  SE  framework  and 
documented  trending  information  for  further  framework  updates  and  conclusions. 


Following  the  tool  development  and  case  study  application,  the  SE  framework  tool  was 
delivered  to  a  group  of  prominent  systems  engineers  both  internal  and  external  to  AFRL 
with  the  goal  of  validating  the  results  of  the  tool  for  various  project  states  as  well 
gathering  impressions  and  feedback  with  respect  to  the  tool’s  functionality  and  ease  of 
navigation.  The  validators  were  asked  to  provide  specific  feedback  in  the  following 
areas:  Functionality,  Activity  Descriptions,  Tailoring,  References  and  Tools,  and  General 
Comments.  Feedback  from  the  validators  added  final  refinements  to  the  SE  framework 
and  is  presented  in  Chapter  IV. 
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IV.  Analysis  and  Results 


The  SE  and  Project  Taxonomies  provided  the  foundation  for  an  SE  tailoring  framework. 
The  research  team’s  goal  of  delivering  an  output  that  will  be  utilized  by  the  S&T 
community  drove  the  need  for  a  validated,  user-friendly  tailoring  tool,  based  on  real- 
world  projects  that  successfully  applied  SE  principles.  The  initial  tailoring  effort  used  the 
case  studies  to  correct  any  false  assumptions  and  thought  processes.  The  tool  validation 
effort  provided  several  independent  views  of  the  results  and  generated  areas  for 
immediate  adjustment  as  well  as  future  work. 

SE  Tailoring  Tool  Development 

A  simple  user  interface  was  deemed  critical  to  the  success  of  the  tailoring  tool.  The  team 
envisioned  drop  down  boxes  with  selectable  domain  values,  desired  processes  and  detail 
levels,  and  easy  to  navigate  controls.  The  course  to  this  interface  wandered  through 
explorations  into  using  Microsoft  Access,  Java,  SQL,  and  Visual  Basic.  Ultimately,  a 
simple  solution  was  found  in  Excel  itself,  by  creating  an  interface  worksheet  that  allows 
users  to  select  their  project  domain  values,  and  then  selecting  the  next  worksheet  for  the 
tailoring  results.  This  decision  made  the  assumption  that  users  would  have  access  to 
Excel  2007  and  would  be  familiar  enough  with  Excel  to  click  between  worksheets  and 
perform  simple  grouping  and  filtering  functions  (if  desired).  The  number  of  SE 
activities  (over  1,200)  was  still  fairly  cumbersome  and  intimidating,  so  the  team 
implemented  the  grouping  feature  in  Excel  2007,  which  uses  a  collapsible  “+/-“  system 
(similar  to  MS  Project)  to  allow  users  to  drill  down  to  the  desired  detail  level. 
Additionally,  filtering  is  enabled  on  the  worksheet  to  allow  users  to  select  only  specific 
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activities,  detail  levels,  or  even  weight  scores.  The  SE  Tailoring  Tool  User’s  Guide  at 
Appendix  B  contains  specific  details,  operating  instructions,  and  guidance  to  modify  the 
tool. 

Following  the  first  case  study  (HELLTP,  described  in  detail  below),  the  team  revised  the 
tailored  weight  calculations.  The  HELLTP  case  study  reinforced  suspicions  that  the 
Integration  Level  and  Requirements  Maturity  played  a  minor  role  in  SE  tailoring,  and  that 
Project  Budget  was  the  dominant  discriminant  in  what  SE  activities  a  project  lead  would 
accomplish.  Analysis  of  the  tailored  weight  data  (see  Appendix  C)  backs  up  this 
assessment  and  bolsters  the  argument  for  using  discriminant  weight  factors.  To  this 
point,  the  tailored  weight  calculation  treated  each  discriminant  as  equal,  so  if  all  six 
discriminant  categories  were  used,  each  discriminant  would  have  a  16.67%  factor  in  the 
tailoring  score.  The  tailoring  tool  was  changed  to  provide  a  weight  factor  for  each 
discriminant.  Project  Budget  was  assigned  a  30%  factor,  RDT&E  Category,  TRL,  and 
Core  Process  were  each  assigned  20%  factors,  and  Integration  Level  and  Requirements 
Maturity  were  assigned  5%  factors  (for  a  total  of  100%).  This  gives  the  tailoring  tool  a 
greater  spread  of  weight  values  (as  noted  in  case  study  feedback)  and  provides  a  more 
realistic  view  of  how  users  should  assess  whether  or  not  to  apply  specific  SE  activities.  It 
is  important  to  note  that  the  discriminant  weight  factors  can  be  easily  changed  if  a 
specific  tool  user  wishes  to  customize  it  in  the  future. 

To  accommodate  projects  that  don’t  have  a  single  identifiable  domain  value  for  each 
discriminants,  the  tool  will  maintain  a  100%  scale  for  tailoring  recommendations  for  non- 
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standard  program  inputs.  If  no  domain  values  are  selected  for  a  discriminant,  that 
discriminant’s  weight  factor  will  be  proportionally  distributed  among  the  other 
discriminants  that  do  apply.  Alternately,  if  multiple  domain  values  are  selected  for  a 
given  discriminants,  that  discriminant’s  weight  factor  is  split  equally  between  the  number 
of  selected  domain  values,  so  the  other  discriminants’  weight  factors  do  not  change. 


Tailored  Weight  Analysis 

The  SE  Taxonomy  resulted  in  350  SE  activities  with  tailored  weights  at  Level  3.  The 
team  performed  evaluations  on  the  amount  of  tailoring  for  various  project  types,  as  well 
as  the  impact  of  the  weight  factors  described  above.  Three  distinct  project  types  were 
evaluated  for  tailoring  (Table  5). 


Table  5:  Project  Domain  Values  for  Statistical  Analysis 


Discriminant 
/  Project  # 

RDT&E 

Category 

Budget 

Core 

Process 

TRL 

Integration 

Level 

Rqmts 

Maturity 

1 

6.1 

<$500K 

CP-1 

1-2 

Subsystem 

Tech  Push 

2 

6.2 

$500K  -  $2M 

CP-2 

3-4 

System 

Tech  Push 

3 

6.3 

>$2M 

CP-3 

7-9 

Mission 

Rqmts  Pull 

Additionally,  each  type  of  project  was  evaluated  with  three  different  weight  factor 


schemes  (Table  6). 


Table  6:  Weight  Factor  Schemes 


Discriminant 
/  Project  # 

RDT&E 

Category 

Budget 

Core 

Process 

TRL 

Integration 

Level 

Rqmts 

Maturity 

Current 

20% 

30% 

20% 

20% 

5% 

5% 

Middle 

20% 

20% 

20% 

20% 

10% 

10% 

Equal 

16.66% 

16.66% 

16.66% 

16.66% 

16.66% 

16.66% 

Histograms  of  the  Level  3  weights  for  each  project  type  and  weight  factor  are  in  Figures 
6-8  below. 
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Figure  6:  SE  Activity  Weight  Histogram  for  Project  1 


Tailoring  for  6.2,  $500K-$2M,  CP-2,  TRL  3-4,  System 
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Figure  7:  SE  Activity  Weight  Histogram  for  Project  2 
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Figure  8:  SE  Activity  Weight  Histogram  for  Project  3 


The  data  clearly  shows  that  there  is  more  tailoring  for  smaller,  less  mature  projects  than 
for  larger,  more  mature  projects.  Only  39%  of  the  activities  in  Project  1  carry  100% 
weights,  while  65%  of  the  activities  in  Project  2  and  95%  of  the  activities  in  Project  3 
carry  100%  weights.  The  weight  factors  prove  to  have  minimal  impact  on  the  tailoring 
for  most,  but  not  all  projects.  There  is  almost  no  impact  from  the  various  weight  factors 
in  Project  3  -  only  3%  of  the  activities  changed  weights  using  the  different  factors.  The 
factor  selection  played  a  more  prominent  role  in  Project  1,  where  about  45%  of  the 
activities  had  their  tailored  weight  scores  affected  by  10-20%  based  on  the  factor  criteria. 


Case  Study  Summaries 

After  completing  the  initial  development  of  the  tailored  SE  framework,  the  research 
investigated  six  case  studies  based  on  recent  S&T  projects  that  valued  systems 
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engineering  principles.  The  research  identified  the  case  study  projects  by  their  domain 
values  in  the  project  taxonomy,  reviewed  documentation,  and  interviewed  knowledgeable 
points  of  contact  to  resolve  gaps  and  discrepancies  in  documentation.  After  the  case 
study  reviews,  the  tailored  SE  framework  was  updated  to  reflect  the  lessons  learned.  The 
case  study  reviews  sought  out  only  application  and  impact  of  SE  processes,  methods  and 
tools.  Though  specific  information  about  the  project  technology  and  operational  use  was 
often  available,  it  was  not  the  primary  focus  of  this  research  and  is  not  included  in  this 
report. 

The  research  completed  case  study  reviews  by  assembling  a  representative  tailored  SE 
framework  output  based  on  the  applicable  project  domain  values.  The  tailored  output 
was  at  the  “3”  level  to  allow  a  comprehensive  review  of  significant  activities  while  not 
subjecting  the  participants  to  a  1,200+  item  survey  for  each  case  study.  The  provided 
project  documentation  either  did  or  did  not  support  evidence  of  each  SE  activity,  which 
was  annotated  on  a  review  sheet.  Any  SE  activities  with  an  absence  or  conflicting 
documentation  evidence  were  posed  as  interview  questions  for  the  point  of  contact  for 
further  clarification. 

The  initial  list  of  potential  case  studies  contained  10  candidate  projects.  Seven  candidates 
came  from  across  AFRL,  two  were  independent  research  and  development  projects  from 
defense  contractors,  and  one  was  from  NASA.  Ultimately,  six  project  points  of  contact 
were  responsive  enough  to  provide  the  team  with  a  thorough  case  study  opportunity 
(Table  4).  The  predominant  case  study  findings  identified  the  need  to  establish  a 
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project’s  context  and  time  horizon,  refined  the  SE  taxonomy,  improved  the  SETT-STP 
tool  functionality,  and  most  significantly,  revealed  a  clear  disconnect  on  SE  terminology 
and  applicability  for  6.1  (Basic  Research)  projects.  Summaries  of  each  case  study  review 
follows. 


Case  Study  1:  High  Energy  Laser  on  a  Large  Tactical  Platform  (HELLTP) 

The  HELLTP  project  came  to  the  team  by  recommendation  from  the  AFRL  Systems 
Engineering  Council  during  the  initial  project  briefing  in  September  2008.  This  multi¬ 
directorate  systems  engineering  initiative  implemented  the  IPPD  process  for  the  three- 
phased  project,  conducted  from  2005  to  2008.  In  his  response  to  the  “Case  Study  Pre- 
Survey  Questionnaire”,  the  project  subject  matter  expert  provided  the  project 
discriminants  for  HELLTP  as  follows:  RDT&E  Category:  6.2,  RDT&E  Budget:  Greater 
than  $2M,  Core  Process:  CP-2,  TRL:  5-6,  Integration  Level:  Subsystem,  Requirements 
Maturity:  Requirements  Pull.  Documentation  review  consisted  of  the  “Task  1  Final 
Report”  (September  2006),  the  “Thermal  Management  System  Analysis  for  the  Airborne 
Advanced  Electrical  Laser  System”  (March  2007),  and  the  “Final  Report  for  the  Tactical 
Laser  Characterization  and  Integration  Study”  (September  2008). 
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The  research  team  combed  through  these  documents  to  determine  which  SE  processes 
within  the  SE  taxonomy  the  HELLTP  project  implemented,  and  focused  on  annotating 
what  was  done  on  the  project  and  not  specifically  on  how  well  it  was  done.  Results  were 
recorded  by  labeling  each  of  the  Level  3  SE  processes  according  to  the 
“YES/NO/SOME/NO  DATA”  criteria  established  by  the  methodology.  This  review 
resulted  in  38  project  questions  for  the  subsequent  interview,  as  well  as  inconsistencies 
within  the  tool  itself,  such  as  areas  where  the  tailoring  inputs  and  calculations  were 
incorrect  or  missing  and  where  related  tasks  were  weighted  differently  within  the  SE 
Framework  output.  The  documentation  review  also  highlighted  the  need  to  include  a 
Level  3  task  and  associated  weightings  within  TP-2  (Logical  Analysis)  to  “identify 
training  requirements”  for  a  project.  The  team  implemented  these  changes  prior  to  the 
review  of  the  next  case  study. 

The  follow-up  interview  covered  the  38  questions  from  the  documentation  review  and  led 
into  a  discussion  regarding  the  current  interface  and  functionality.  The  questions  focused 
on  annotating  the  SE  processes  within  the  framework  where  supporting  documentation 
was  inconsistent  or  incomplete.  Additional  interview  discussions  highlighted  the 
importance  of  correctly  establishing  a  project  context  for  the  “state”  determination  and 
review  process,  as  well  as  the  importance  of  first-hand  project  knowledge  in  correctly 
identifying  which  SE  tasks  were  accomplished.  The  interview  also  identified  possible 
functionality  improvements  or  changes  to  the  tool  itself. 
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Case  Study  2:  Powered  Low  Cost  Autonomous  Attack  System  (PLOCAAS) 

The  PLOCAAS  project  evaluated  mission  concepts,  defined  performance  objectives, 
investigated  environmental  constraints,  and  evaluated  candidate  sensing  technologies  for 
a  powered  version  of  a  low-cost  searching  weapon  system.  The  project  was  conducted  in 
the  late  1990s  by  what  was  then  known  as  the  Munitions  Directorate  of  AFRL.  The 
subject  matter  expert  was  a  former  program  manager  for  the  project  and  served  as  the 
point  of  contact  for  the  case  study.  He  provided  the  project  discriminants  for  PLOCAAS 
as  follows:  RDT&E  Category:  6.3,  RDT&E  Budget:  Greater  than  $2M,  Core  Process: 
CP-2,  TRL:  7-9,  Integration  Level:  System,  Requirements  Maturity:  Technology  Push. 

PLOCAAS  focused  on  early  concept  and  technology  development,  and  as  such,  applied  a 
majority  of  its  systems  engineering  effort  on  the  early  technical  processes.  To  varying 
levels  of  formality,  the  project  accomplished  most  of  the  activities  in  TP-1  through  TP-7. 
The  project  did  not  transition  to  another  developing  or  using  organization,  so  most  of  the 
TP-8  activities  did  not  apply  to  the  project.  The  technical  management  processes  were  all 
addressed,  but  the  program  did  not  apply  these  activities  robustly  across  the  board. 
Significant  diligence  was  applied  in  the  decision  analysis  and  technical  assessment 
processes.  The  project  applied  minimal  configuration  management  and  thinly 
documented  project  requirements,  risks,  and  interfaces.  Other  technical  management 
processes  were  generally  applied  at  a  high  level,  but  specific  activities  were  omitted  or 
accomplished  informally  [Jacques,  2008], 

Following  the  documentation  review,  the  team  conducted  a  follow-up  interview, 
according  to  the  established  process.  The  subject  matter  expert  evaluation  provided  by 
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the  point  of  contact  proved  to  be  much  more  thorough  than  the  student  review  given  the 
scope  and  detail  level  of  the  documentation.  The  interview  prompted  several  revisions  to 
the  SE  framework,  including  taxonomy  overhauls  for  many  of  the  technical  management 
processes.  Some  specific  processes  were  merged  (the  definitions  of  threshold  versus 
objective  performance  parameters)  and  reworded  (the  use  of  “interface  architecture” 
preferred  by  Buede  was  changed  to  “interface  control  methods”.)  Additionally,  the  team 
found  it  useful  to  insert  comment  boxes  into  the  SE  tool  to  provide  definitions  or 
clarifying  statements  to  the  SE  activities. 

Case  Study  3:  Layered  Sensing 

The  Layered  Sensing  project  is  the  second  phase  of  a  multi-directorate  effort  designed  to 
“improve  the  quality  and  timeliness  of  acquiring,  sorting,  processing,  and  reporting 
information  to  improve  effects  based  situation  awareness”  [Sensors  Directorate,  ii] .  This 
phase  of  the  project  focused  on  identifying  the  requisite  tools  and  measures  for  building 
an  executable  architecture  designed  to  evaluate  various  sensor  system  combinations.  The 
project  subject  matter  expert,  provided  the  project  discriminants  for  Layered  Sensing  as 
follows:  RDT&E  Category:  6.2,  RDT&E  Budget:  $200K  -  $2M,  Core  Process:  CP-2, 
TRL:  5-6,  Integration  Level:  Mission,  Requirements  Maturity:  Technology  Push. 
Documentation  review  consisted  of  the  Phase  II  Study  Plan  and  its  associated  annexes. 

Similar  to  the  previous  projects,  the  team  reviewed  the  documentation  for  indications  of 
SE  activities,  processes,  and  tools.  This  review  resulted  in  nine  specific  questions  for  the 
follow-up  interview,  but  also  left  over  250  of  the  Level  3  activities  unresolved,  further 
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justifying  the  conclusion  from  the  HELLTP  and  PLOCAAS  case  studies  to  have  a  subject 
matter  expert  or  project  representative  involved  in  determining  which  activities  were 
accomplished.  A  thorough  review  of  the  SE  tailoring  tool  output  with  the  subject  matter 
expert  on  30  December  2008  reconciled  the  specific  questions  and  resolved  the  gaps  from 
the  documents,  but  also  pointed  out  inconsistencies  in  the  level  of  detail  within  the  SE 
taxonomy  hierarchy,  the  need  to  further  tailor  activities  in  the  “Project  Budget” 
discriminant,  and  the  desire  to  see  a  roll-up  of  the  Level  3  tailored  weights  at  Levels  1 
and  2. 

The  primary  inconsistency  revolved  around  the  level  of  detail  for  Level  3  activities 
between  the  TPs  and  the  TMPs,  where  some  of  the  TMP  Level  3  activities  were  much 
more  specific  than  those  of  the  TPs.  This  feedback  resulted  in  moving  many  of  the  TMP 
Level  3  tasks  (particularly  within  TMP-2  “Technical  Planning”  and  TMP-3  “Technical 
Assessment”)  down  to  Level  4  in  the  SE  taxonomy,  simplifying  the  user  interface  for 
those  processes.  The  case  study  review  also  uncovered  a  need  to  further  tailor  the  Total 
Budget  discriminant  for  the  “$200K  to  $2M”  domain  value  in  TMP-6  “Configuration 
Management”  and  TMP-7  “Technical  Data  Management”,  as  many  of  these  activities 
were  given  a  “100%”  weight,  but  seemed  to  be  too  heavily  weighted  for  the  scope  of  the 
project.  The  team  reviewed  the  initial  tailoring  effort  and  made  changes  where 
appropriate,  but  left  the  tailoring  intact  where  the  activities  seemed  critical  to  project 
success.  The  final  change  to  the  SE  tailoring  tool  was  the  addition  of  an  indication  at 
Levels  1  and  2  of  the  range  of  tailored  weights  for  the  subordinate  Level  3  tasks. 
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Another  critical  result  from  this  case  study  was  the  importance  of  framing  the  context  of 
a  project  when  selecting  the  discriminant  domain  values  in  the  SE  tailoring  tool.  As  the 
Layered  Sensing  project  consists  of  multiple  phases,  the  total  budget  for  the  current  phase 
does  not  correctly  capture  the  scale  of  the  overall  project,  thus  potentially  reducing  the 
tailored  weights  for  activities  that  would  be  valuable  within  the  larger  context.  This  was 
notable  in  the  project  interview,  where  the  subject  matter  expert  indicated  that  although 
certain  SE  activities  were  not  accomplished  on  the  current  phase  (largely  concept 
exploration  with  little  implementation),  they  would  be  beneficial  in  future  phases  of 
Layered  Sensing. 

Case  Study  4:  Northrop  Grumman  Internal  Research  and  Development 
The  research  sought  insight  into  SE  activities  within  a  corporate  project  to  determine 
whether  the  SE  taxonomy  was  appropriate  for  an  S&T  environment  outside  of  DoD,  as 
well  as  to  gain  perspective  on  how  SE  was  implemented  when  not  constrained  by 
government  contract  requirements.  At  the  team’s  request,  The  Northrop  Grumman  (NG) 
Corporation’s  Integrated  Systems  Division  provided  an  Internal  Research  and 
Development  (IRAD)  project  for  a  case  study  review.  While  the  project  is  ultimately 
targeted  for  fielding  in  the  defense  environment,  it  was  not  governed  by  a  government 
contract  and  thus  was  not  subject  to  government  systems  engineering  control.  For 
purposes  of  protecting  Northrop  Grumman’ s  competitive  interests,  details  of  the  project 
will  not  be  provided  in  this  report.  However,  the  technical  details  are  not  important  to  the 
systems  engineering  analysis  that  was  performed.  The  NG  project  fit  the  following 
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domain  values:  6.3,  >$2  million,  CP-2,  TRL  5-6,  Subsystem  Level,  and  Technology 
Push. 

Due  to  the  sensitive  nature  of  the  NG  project,  the  team  did  not  receive  project 
documentation.  Instead,  the  team  sent  the  latest  draft  of  the  SE  Tailoring  Framework  to 
NG,  who  had  their  project  team  go  through  the  activities  in  the  SE  taxonomy.  The  NG 
personnel  indicated  whether  or  not  they  performed  each  activity  in  the  project’s 
execution.  They  also  made  comments  about  the  tailored  weight  levels  and  the  tool’s 
usability.  After  receiving  NG’s  comments,  the  student  team  formulated  a  set  of  interview 
questions;  nine  about  why  activities  were  or  were  not  performed,  as  well  as  six  “big 
picture”  questions  about  NG’s  internal  SE  processes.  The  NG  interview  resulted  in 
several  changes  to  the  framework  activity  descriptions  and  SE  Rigor  scores,  which  were 
incorporated.  The  term  “qualification”  in  TP-8  (Transition)  was  confusing,  and 
ultimately  changed  to  “deliverable”  to  clarify  the  meaning  to  be  a  transition  item  that 
would  undergo  certain  transition  activities.  NG  noted  that  no  manufacturing  process 
improvements  were  made  under  their  IRAD  project  due  to  limitation  of  time  and  funding. 
In  fact,  NG’s  IRAD  projects  don’t  generally  cross  into  the  realm  of  manufacturing  or 
producability;  lab  prototypes  are  used  for  product  evaluations. 

A  few  significant  differences  were  noted  between  NG’s  IRAD  and  Contracted  Research 
and  Development  (CRAD)  efforts.  IRAD  projects  don’t  usually  solicit  bids  for  various 
suppliers;  rather,  they  pick  the  vendor  that  they  know  can  supply  a  product.  Under 
CRAD  rules  (which  are  inherent  to  government  development  activities),  competition 
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between  vendors  and  suppliers  is  mandatory.  Additionally,  NG’s  IRAD  projects 
typically  apply  rigorous  and  detailed  SE  practices,  but  they  are  often  less  formal,  more 
streamlined,  and  more  self-contained  than  for  CRAD.  The  project  leaders  felt  they 
applied  the  right  amount  of  SE  rigor,  which  resulted  in  a  successfully  tested  prototype. 
Generally,  NG  indicated  that  they  did  execute  most  of  the  recommended  SE  activities 
listed  in  the  framework. 

Additional  NG  feedback  indicated  that  the  tailored  weights  did  not  always  match  with 
their  assessment  of  required  SE  rigor.  However,  they  said  the  framework  was  a  good 
exercise  to  remind  project  managers  and  systems  engineers  to  apply  proper  SE  practices. 
NG  suggested  that  applying  inputs,  outputs,  constraints  and  sequencing  to  each  SE 
activity  would  greatly  increase  the  tool’s  value.  This  suggestion  falls  outside  the  scope  of 
this  research  project  and  is  recommended  for  follow-on  work  in  Chapter  V. 

Case  Study  5:  Deployed  Base  Energy  Study 

The  fifth  case  study  is  a  project  planning  study  to  develop  an  investment  strategy  for 
creating  more  efficient  methods  of  providing  energy  to  deployed  airbases.  The 
Deployed  Base  Energy  study  was  conducted  by  AFRL’s  Materials  and  Manufacturing 
Directorate  (AFRL/RX).  This  case  study  was  unique  because  it  only  focused  on  early  SE 
processes  associated  with  determining  requirements,  logical  analysis,  and  making 
decisions  about  what  technologies  to  pursue.  This  activity  did  not  intend  to  deliver  any 
capabilities,  so  it  was  a  good  test  for  the  left-hand  side  of  the  systems  engineering  “Vee”. 
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The  project  domain  values  for  the  Deployed  Base  Energy  study  were:  6.2,  <$500K,  CP- 
3,  TRL  3-9,  System  and  Subsystem  Levels,  and  Requirements  Pull. 

The  Deployed  Base  Energy  study  used  a  self-contained  SE  methodology,  called  Systems 
Engineering  For  Science  and  Technology  (SETFST),  which  is  a  “multi-criteria  analytical 
process  for  comparing  alternatives”  [SynGenics,  2008;  10].  The  SETFST  method 
encompasses  similar  activities  to  the  early  DAG  technical  and  technical  management 
processes.  The  five  SETFST  process  steps  are:  1.  Assemble  an  Integrated  Product 
Team;  2.  Develop  Desirements;  3.  Generate  Alternatives;  4.  Evaluate  Alternatives;  and 
5.  Document.  These  easily  map  to  TP-1  (Requirements  Development),  TP-2  (Logical 
Analysis),  and  TMP-1  (Decision  Analysis).  The  study  intended  to  evaluate  possible 
design  solutions,  not  deliver  a  specific  design,  so  the  rest  of  the  DAG  processes  were  not 
applicable  for  this  phase  of  the  project. 

This  case  study  review  consisted  of  the  point  of  contact’s  assessment  of  how  the 
framework’s  recommended  activities  were  accomplished  via  the  mature  SETFST 
process,  followed  by  a  personal  interview.  The  review  clearly  demonstrated  that  the 
DAG  TP-1,  TP-2,  and  TMP-1  activities  and  weights  were  in  line  with  the  successful 
SETFST  study  results.  The  project  subject  matter  expert  also  closely  evaluated  the  rest 
of  the  processes,  activities  and  weights  in  the  tool  and  assessed  that  they  were  reasonable, 
based  on  his  20+  years  of  systems  engineering  experience.  Specifically,  he  couldn’t  find 
justification  to  change  any  activities  or  weights  within  the  framework.  He  suggested  that 
the  User’s  Guide  should  introduce  the  tailoring  tool  at  a  more  basic  level,  but  liked  the 
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flexibility  of  the  framework.  He  also  noted  that  the  tool  would  be  useful  primarily  to 
users  who  had  a  basic  familiarity  with  systems  engineering  activities,  but  that  the  new 
user  may  struggle  with  some  of  the  terminology  and  intent  contained  within  the 
framework.  This  resulted  in  the  addition  of  a  Glossary  tab  in  the  tool. 

Case  Study  6:  Basic  Research  RXQ  Project 

The  sixth  case  study  came  as  a  direct  result  of  the  team  traveling  to  Tyndall  AFB,  FL  for 
the  Systems  Engineering  Council  Face-to-Face  meeting  to  present  project  status  and  tool 
demonstration  on  2-4  February  2009.  Specifically,  conversations  with  AFRL/RXQ 
during  a  side-session  of  the  meeting  presented  an  opportunity  for  the  team  to  conduct  this 
case  study  with  the  two  project  leads. 

While  presenting  background  information  and  the  requirement  for  this  thesis  effort,  the 
team  outlined  the  8  questions  from  AFRLI  61-104  (not  currently  required  by  the 
instruction  for  6.1  projects).  The  points  of  contact  quickly  recognized  the  questions 
relating  to  AFRL  Form  2913  (Sept  2002),  required  by  AFRLI  61-202  “Laboratory 
Management  Review  (LMR).”  When  asked  about  the  usefulness  of  these  questions,  the 
senior  project  lead  stated  that  filling  out  the  form  was  a  burden  until  the  new  technical 
director  helped  them  see  that  going  through  the  LMR  process  actually  helped  them 
structure  their  projects,  with  the  example  of  translating  the  goal  of  their  basic  research 
into  the  requirement  for  the  project.  An  additional  observation  identified  differences 
between  basic  research  and  systems  engineering.  His  assertion  was  that  SE  drives  design 
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toward  a  known  goal  (i.e.  system),  while  basic  research  is  largely  guided  by  the  data 
presented  from  the  experiment  itself. 

As  the  team  began  the  case  study  review  of  “Characterization  of  the  Nucleation  and 
Binding  Sites  of  Hen-Egg- White-Lysozyme  to  Silica”,  the  research  team  tried  to  get  the 
project  officers  to  give  “Yes/No”  answers  to  the  Level  2  SE  activities.  It  was  quickly 
apparent  that  the  project  officers  needed  the  team  to  translate  each  of  the  SE  activities 
within  the  framework  into  vernacular  more  familiar  to  basic  research  scientists.  In  an 
attempt  to  overcome  this  obstacle,  the  team  explained  the  Level  2  SE  activities  within  the 
framework  until  the  project  officers  understood  the  underlying  value  of  the  activity  and 
replied  with  similar  activities  performed  for  basic  research.  This  approach  proved 
cumbersome,  so  the  team  elevated  the  interview  goal  to  determining  if  the  project  officers 
understood  the  simple  definition  of  the  activity  or  if  it  needed  translation.  Ultimately,  the 
time  and  effort  involved  for  translation  and  explanation  led  the  team  to  abandon  a 
detailed  review  of  the  case  study,  with  obvious  conclusions  in  hand. 

The  team  recognized  that  the  framework  will  have  limited  usefulness  for  the  6.1 
community  as  written.  To  make  the  framework  applicable  to  basic  research  projects,  the 
SE  terminology  must  be  translated  and  adapted  to  more  closely  represent  existing 
scientific  discovery  methods.  Additionally,  an  adjustment  must  be  made  to  the 
framework  to  allow  tailoring  for  a  lower  level  in  the  Integration  Level  discriminant  (for 
instance,  “Component”  or  “Technology”  for  cases  where  a  project  with  a  single 
functional  output  is  desired.  These  modifications,  though  critical  to  implementing  the 
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framework  for  the  6.1  community,  were  realized  too  late  to  be  implemented  under  this 
research  project.  They  are  contained  as  follow-on  project  recommendations  in 
Chapter  V. 

Validation 

To  validate  the  framework,  the  team  provided  a  Beta  version  of  the  Systems  Engineering 
Tailoring  Tool  for  Science  &  Technology  Projects  (SETT-STP)  framework  to  eight 
senior  AFRL  scientists  and  engineers  with  SE  experience.  The  accompanying 
instructions  asked  the  validators  to  review  two  out  of  four  notional  project  types 
(Table  6.)  A  column  in  the  tool  allowed  for  specific  comments  for  each  SE  activity  and 
tailored  level  of  SE  Rigor,  as  well  as  general  comments  based  on  Functionality,  Activity 
Descriptions,  Tailoring,  References  and  Tools,  and  General  Comments.  Six  validation 
responses  were  received  and  grouped  into  several  functional  areas:  SE  Rigor; 
Terminology;  Methodology;  Tool  Liability;  CONOPS;  and  Follow-On  Work.  The 
predominant  results  are  contained  in  the  SE  Rigor,  CONOPS,  and  Follow-On  Work 
areas.  The  team  analyzed  the  responses  for  trends  as  well  as  incorporating  specific 
recommendations  where  possible.  Many  recommendations  were  too  large  in  scope  or 
required  major  changes  to  the  framework  approach  to  be  implemented  prior  to  the 


Table  8:  Mapping  of  Validation  Projects  to  Discriminants 
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framework’s  release.  These  major  recommendations  are  summarized  in  Chapter  V  for 
incorporation  into  future  versions  of  the  framework. 

SE  Rigor  Comments 

Most  of  the  validation  comments  for  the  SE  rigor  percentages  displayed  in  the  tool  were 
applicable  to  Project  B.  The  validators  recognized  that  many  of  the  SE  activities  (as 
written  in  the  framework)  were  not  directly  applicable  to  small,  basic  research  projects 
and  that  the  SE  rigor  values  should  be  reduced,  and  in  many  cases  even  0%.  Although 
many  specific  changes  were  recommended,  the  team  did  not  implement  them  in  the  tool, 
due  to  the  previously  recognized  need  to  translate  and  adapt  the  SE  framework  for  basic 
projects.  Making  detailed  adjustments  to  the  tailoring  values  would  have  little  worth 
when  an  overhaul  of  the  tool  for  6.1  projects  is  a  major  recommendation  from  the 
validation  effort. 

The  only  other  project  to  receive  significant  comments  on  the  SE  rigor  values  was  Project 
C,  where  the  validator  agreed  with  the  framework  output,  which  tailors  in  nearly  all  SE 
activities  at  a  formal  level. 

Terminology  Comments 

The  two  dominant  overarching  comments  (and  several  detailed  specific  comments)  from 
the  validation  phase  reinforced  conclusions  drawn  from  the  Basic  Research  case  study. 
Specifically,  the  existing  framework  terminology  for  SE  activities  generally  does  not 
apply  for  6.1  (Basic  Research)  projects.  Additionally,  a  fourth  domain  value  under  the 
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Integration  Level  discriminant  is  required  to  describe  single  component  or  technology 
development  activities.  The  research  team  strongly  agrees  with  these  comments,  but  was 
unable  to  implement  them  in  the  framework  under  the  scope  of  this  thesis.  These 
changes  are  recommended  in  the  follow-on  work  section  in  Chapter  V. 

Additional  explanation  is  needed  to  instruct  users  that  the  selected  TRL  domain  value 
should  apply  to  the  desired  TRL  end  state  for  a  given  phase  of  a  project.  This  change 
was  made  to  the  SETT-STP  user  guide  and  described  in  the  methodology  section  of  the 
thesis  report. 

Methodology  Comments 

The  validation  effort  revealed  suggestions  about  the  number  and  nature  of  discriminant 
categories  used  in  the  project  taxonomy  that  resulted  in  the  activity  tailoring  results.  One 
suggestion  was  to  re-evaluate  the  discriminants  to  better  define  a  fewer  number  of  factors 
that  are  integral  to  recommending  SE  rigor  for  a  project.  The  six  discriminants  that  are  in 
the  framework  were  developed  as  a  direct  reflection  of  how  AFRL  manages  its  projects 
today.  The  student  team  considered  an  initial  approach  for  the  project  taxonomy  that 
used  just  two  overarching  discriminants  (Project  Complexity  and  Project  Maturity).  This 
approach  was  discarded,  as  it  did  not  provide  enough  fidelity  to  capture  the  broad  range 
of  AFRL  project  types  that  could  benefit  from  the  tailoring  framework.  Note  that  a 
framework  user  can  set  up  the  tool  to  incorporate  just  one  or  two  discriminants  and  still 
obtain  proper  tailoring  recommendations. 
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A  different  suggestion  was  that  using  Core  Processes  as  a  discriminant  might  indicate 
that  some  customers  deserve  more  SE  than  others.  This  is  an  accurate  assessment  based 
on  the  ability  of  the  customer  organization  to  add  additional  SE  rigor  to  the  ultimate  end 
product.  For  instance,  a  CP-3  project  delivers  to  an  operational  user  who  can’t  perform 
additional  integration  or  data  management  planning,  whereas  a  CP-2  project  typically 
delivers  to  a  Systems  Program  Office  who  will  have  formal  SE  processes  in  place  to 
ensure  the  final  deliverable  is  matured  by  the  maximum  possible  level  of  SE  rigor.  Thus, 
an  AFRL  CP-3  project  should  apply  more  SE  rigor  than  a  CP-2  project.  There  should  be 
no  interpretation  on  the  level  of  importance  of  one  customer  over  another  based  on  this 
distinction. 

Another  suggestion  was  that  the  RDT&E  Budget  Category  discriminant  should  not  affect 
the  framework’s  recommended  tailored  SE  rigor  level.  The  team  disagrees  with  this 
assessment,  as  the  vast  differences  between  a  6.1  project  and  a  6.3  project,  as  noted  in  the 
case  studies,  are  enough  to  drive  an  overhaul  to  the  terminology  and  application  of  SE 
principles  based  solely  on  the  type  of  research  project  (6.1  vs.  6. 2/6. 3). 

A  fourth  suggestion  was  that  the  Integration  Level  is  a  more  important  discriminator  and 
driver  of  SE  rigor  than  TRL.  The  tailoring  results  and  accompanying  sensitivity  analysis 
contained  in  Appendix  C  show  that  Integration  Level  and  Requirements  Maturity  had 
almost  no  effect  on  SE  rigor  tailoring  values,  so  no  changes  were  made  based  on  this 
suggestion. 
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The  final  methodology  suggestion  was  to  include  more  than  three  Project  Budget  domain 
values.  The  project  taxonomy  originally  included  four  Project  Budget  domain  values,  but 
was  consolidated  down  to  three  due  to  a  lack  of  difference  in  tailoring  results  between  the 
top  two  categories.  This  approach  would  be  useful  if  specific  metrics  could  be  captured 
on  historic  projects  to  provide  additional  tailoring  insight. 

Overall,  none  of  the  methodology  comments  resulted  in  changes  to  the  final  framework. 
However,  some  of  the  suggestions  may  be  applicable  to  a  follow-on  tailoring  effort. 

Tool  Usability  Comments 

Members  of  the  AFRL  SE  Council  provided  usability  feedback  during  individual  and 
group  presentations.  Specific  Council  members  and  validators  commented  favorably  on 
the  tool’s  functionality  and  navigation  ease,  to  the  point  of  requesting  the  tool  for  their 
immediate  use.  Another  comment  praised  the  SE  taxonomy,  indicating  the  best  use  of 
the  tool  is  in  identifying  a  comprehensive  set  of  SE  activities  to  be  accomplished  by  S&T 
projects. 

CONOPS 

Several  suggestions  were  made  as  to  how  AFRL  should  apply  the  framework.  None  of 
these  comments  resulted  in  changes  to  the  tool,  but  they  are  included  for  AFRL’s  future 
consideration.  One  suggestion  was  that  the  framework  should  be  managed  and 
implemented  at  the  highest  possible  level  within  AFRL,  thus  increasing  the  chances  of 
the  tool  improving  SE  coherence  across  Technology  Directorates.  The  final  comment  re¬ 
iterated  the  thought  that  the  framework,  as  currently  written,  does  not  cover  the  6.1 
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(Basic  Research)  end  of  the  S&T  project  spectrum,  due  to  the  challenges  with  relating  SE 
concepts  and  terminology  to  the  pure  science  community. 

Follow-On  Work  Comments 

Several  comments  recommended  major  changes  to  the  SETT-STP  framework  that  were 
not  incorporated  as  part  of  this  research.  The  need  for  a  6.1-specific  translation  and 
inclusion  of  a  “component”  value  in  the  Integration  Level  discriminant  were  previously 
discussed. 

A  “high-medium-low”  construct  for  SE  rigor  was  recommended.  The  idea  behind  this  is 
that  ranges  of  tailored  SE  rigor  percentages  could  be  grouped  into  a  simpler  color-coded 
scheme  that  indicates  whether  the  level  of  SE  rigor  should  be  high,  medium,  or  low.  This 
could  also  represent  the  recommended  SE  taxonomy  activity  detail  level  a  project  should 
follow.  Notionally,  “low”  rigor  activities  should  be  limited  to  Level  2,  “medium”  will 
apply  Level  3  activities,  and  “high”  rigor  categories  should  look  at  Levels  4  and  5.  One 
concern  with  this  is  that  not  all  Level  3  activities  have  subtended  Level  4  and  5  activities. 
This  construct  was  not  implemented,  but  a  description  of  SE  rigor  percentages  and 
appropriate  notional  interpretations  were  included  in  Table  3  and  in  the  User’s  Guide. 

A  suggestion  was  made  to  perform  metrics  collection  on  all  AFRL  projects  that  will 
validate  future  tailoring  values  and  improve  the  framework  for  further  utilization.  This 
could  possibly  be  implemented  with  an  Excel  macro  to  record  parameter  values  and  user 
comments.  The  research  team  applauds  this  suggestion  but  found  that  discerning 
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specific,  quantifiable  metrics  from  historical  and  even  existing  projects  was  difficult.  If 
this  can  be  automated  in  the  future,  it  will  greatly  assist  AFRL  in  appropriately  applying 
it  resources  toward  successful  SE. 

Additional  AFRL-specific  information  and  tailoring  for  the  “Tools”  column  in  the  tool 
would  provide  value  to  project  leads.  To  accomplish  this,  a  focused  effort  across  AFRL 
will  need  to  update  the  “Tools”  column  with  AFRL-specific  tools  to  augment  the 
generally  accepted  tools  that  the  research  identified  in  the  framework.  Tailoring  of 
specific  tools  to  specific  projects  may  require  that  the  number  of  discriminants  is  reduced 
to  ensure  a  feasible  implementation. 
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V.  Conclusions  &  Recommendations 


The  Air  Force  is  continuing  its  efforts  to  implement  systems  engineering  principles 
earlier  in  the  life  cycle  of  research  and  development  projects.  Proper  systems  engineering 
enables  projects  to  meet  cost,  schedule,  and  performance  objectives.  Current  guidance 
clearly  states  the  need  for  early  SE  [Pre  MS-A,  2008;  DSB,  2008]  and  recent  studies 
within  the  Air  Force  Research  Laboratories  [TASE,  2006;  IRT,  2008]  indicate  isolated 
elements  of  successful  SE.  The  recent  implementation  of  the  Focused  Long  Term 
Challenges  (FLTCs)  within  AFRL  provides  a  unique  structure  to  further  employ  SE 
trades  at  the  mission,  system,  and  subsystem  and  component  levels.  Additionally, 
AFRL's  current  operating  instructions  take  a  critical  first  step  towards  challenging 
project  managers,  scientists,  and  engineers  to  consider  SE  principles  for  the  execution  of 
their  projects.  These  principles,  based  on  the  eight  Technical  Processes  (TPs)  and  eight 
Technical  Management  Processes  (TMPs)  from  the  Defense  Acquisition  Guidebook 
(DAG),  are  embodied  in  the  eight  questions  in  AFRLI  61-104.  Additionally,  it  is  clear 
that  the  AFRL  Systems  Engineering  Council  is  making  strides  towards  sharing  and 
implementing  SE  best  practices  between  the  technology  directorates.  If  additional  gains 
towards  cohesive  SE  within  AFRL  are  to  be  achieved,  subsequent  updates  to  operating 
instructions  must  include  a  set  of  common  and  manageable  practices  and  tools  that  take 
the  existing  eight  questions  to  the  next  level  of  SE  rigor.  Implementing  the  SETT-STP 
Framework  will  take  a  large  step  toward  AFRL  realizing  its  goal  of  systems  engineering 
excellence. 
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This  research  developed  a  framework  which  incorporates  a  taxonomy  of  the  SE  activities 
embodied  in  the  DAG,  the  International  Council  for  Systems  Engineering  (INCOSE) 
Handbook  [INCOSE,  2006],  and  “The  Engineering  Design  of  Systems:  Models  and 
Methods”  by  Dennis  Buede  [Buede,  1999].  Although  it  was  not  a  primary  objective  of 
the  research,  the  SE  Taxonomy  was  cited  by  several  interested  parties  to  be  a  valuable 
stand-alone  by-product.  The  SE  Taxonomy  provides  a  comprehensive  list  of  SE 
activities  that  are  functionally  and  hierarchically  organized,  with  the  capability  to  sort  to 
desired  detail  levels.  Likewise,  the  Project  Taxonomy  sets  the  foundation  for  describing 
a  project’s  state,  and  is  not  limited  to  DoD  terminology.  This  taxonomy  was  refined 
through  a  rigorous  evaluation  of  case  studies  and  validation  reviews.  Additionally,  the 
customizable  nature  of  the  Project  Taxonomy  allows  it  to  be  adapted  to  meet  any  S&T 
organization’s  needs. 

The  framework  includes  a  methodology  for  tailoring  the  specific  SE  activities  for  a 
unique  project  state,  based  on  common  discriminants  and  domain  values  currently  found 
within  AFRL.  The  tailoring  applies  a  combination  of  engineering  assessment  and 
numerical  analysis  that  results  in  weight  factors  for  each  project  discriminant  as  they 
affect  an  independent  assessment  of  SE  activity  applicability.  The  product  from  this 
framework,  the  SETT-STP  tool,  is  intended  as  guidance  for  the  amount  of  relative  SE 
rigor  to  apply  for  each  SE  activity  on  a  given  project.  The  tool  uses  accepted  SE 
principles  and  is  designed  to  augment  existing  AFRL  policies  and  practices.  If  properly 
utilized,  SETT-STP  will  allow  scientists  and  engineers  to  simply  input  their  project’s 
state  descriptors  and  receive  as  output  a  comprehensive  set  of  SE  activities  and  their 
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recommended  rigor  that  will  enable  a  final  deliverable  product  ready  for  successful 
transition  to  the  project’s  customer  (Figure  9). 
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-AFRLI  61-104 
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SE  Processes,  Methods  &  Tools 
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Figure  9:  SETT-STP  Functional  Diagram 


The  onus  is  on  each  project  manager  within  their  specific  management  structure  to 
interpret,  adapt,  and  even  modify  the  tailoring  recommendations  to  best  suit  the  needs  of 
their  project.  The  tailoring  recommendations  from  SETT-STP  must  be  evaluated  within 
the  project  context  and  should  not  be  taken  as  a  directive  for  specific  implementation. 
Heuristically,  larger  programs  need  to  execute  a  greater  number  of  SE  activities  to  a 
greater  level  of  formality  and  small  projects  are  not  relieved  of  their  responsibility  to 
apply  appropriate  SE  rigor.  Feedback  from  the  case  studies  and  interactions  with  the 
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AFRL  SE  Council  indicate  SETT-STP  appropriately  establishes  a  framework  for 
approaching  what  SE  activities  are  suitable.  Specifically,  the  SE  Taxonomy  compiles 
proper  SE  activities  for  S&T  projects  in  a  hierarchical  manner,  which  in  turn  facilitates 
the  tailoring  of  those  activities  to  a  specific  project  state.  Finally,  SETT-STP  is 
applicable  to  any  development  project  (laboratories,  program  offices,  and  commercial 
developments,  and  maybe  even  humanity’s  grand  challenges  [NAE,  2008])  and  allows 
scientists,  technologists,  engineers,  and  project  managers  the  opportunity  to  drill  down 
through  the  activities  and  consider  whether  they  are  appropriate  for  a  specific  project. 

Specific  Recommendations 

The  research  revealed  several  opportunities  for  AFRL  to  consider  in  strengthening  the  SE 
program  implementation. 

1 .  AFRL  should  continue  to  emphasize  the  utilization  of  the  16  DAG  processes 
as  a  common  reference.  AFRL  made  a  critical  step  in  this  direction  with  the  8 
questions  in  AFRLI  61-104,  including  a  gross  mapping  to  the  DAG  processes  in 
Attachment  1  [S&T  SE,  9-18].  Utilizing  this  established  and  widely  recognized 
document,  in  conjunction  with  the  SE  Taxonomy  hierarchy  implemented  in 
SETT-STP,  provides  the  opportunity  to  mature  the  project  question  matrix  and  the 
AFRLI  61-104  Attachment  1  correlation  to  the  DAG. 

2.  AFRL  should  increase  visibility  of  SE  activities  within  the  FLTC  construct. 

By  tailoring  reportable  SE  activities  at  the  Challenge,  Problem,  Attribute,  Product, 
and  Program  level,  AFRL  could  bolster  systems  engineering  implementation 
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within  the  FLTCs.  While  this  visibility  of  SE  activities  may  be  perceived  as  an 
additional  burden,  standardizing  the  review  process  to  highlight  achievement  of 
tailored  activities  within  the  FLTCs,  Core  Processes,  and  Technology  Directorates 
(i.e.  everyone  reviewing  the  same  criteria)  will  ultimately  streamline  the  amount 
of  reportable  and  inspectable  information. 

3.  AFRL  should  use  training  and  mentorship  to  foster  a  culture  of  “Systems 
Thinking.”  Scientists  and  engineers  must  be  able  to  recognize  systems 
engineering  principles  in  order  to  correctly  implement  them.  The  SE  case  studies 
demonstrated  that  the  thought  process  behind  systems  engineering  occurs  more 
than  is  commonly  realized.  Putting  more  formal  and  informal  attention  toward 
recognizing  SE  activities  in  everyday  work  will  increase  the  acceptance  of  the 
“SE  mindset”  and  promote  a  receptive  culture  that  will  lead  to  more  proper  and 
rigorous  SE  implementation. 

4.  AFRL  should  consolidate  and  streamline  its  project  management  structure 

as  well  as  systems  engineering  initiatives.  AFRL  manages  projects  in  several 
manners,  as  discussed  when  compiling  the  framework’s  Project  Taxonomy. 

While  each  of  the  structures  has  merit  and  provides  benefit  to  the  project  planning 
and  execution  process,  the  structures  often  interfere  with  each  other  and  hamper 
systems  engineering  and  integration  success.  Likewise,  SE  initiatives  under  the 
purview  of  individual  technology  directorates  each  provide  some  benefit,  but 
result  in  a  fractured  and  inefficient  overall  approach  to  improving  SE  across 
AFRL.  Picking  the  best  practices  and  expanding  them  in  a  smart,  integrated 
manner  will  provide  the  most  effective  SE  value  for  all  S&T  projects. 
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5.  AFRL  should  begin  using  the  SETT-STP  framework  to  guide  SE  efforts 
within  the  Technology  Directorates.  This  research  was  built  on  existing  DoD 
and  AFRL  SE  policy  and  guidance.  It  was  validated  by  S&T  projects  with 
reported  SE  success.  The  case  studies,  incorporated  primarily  at  the 
recommendation  of  the  SE  Council,  spanned  multiple  directorates  within  AFRL, 
each  with  unique  approaches  and  practices  that  validated  the  SE  activities  listed  in 
the  framework.  As  the  SE  Council  continues  to  explore  the  benefits  of  various 
approaches  to  implement  SE  across  the  Technical  Directorates,  it  should  build 
upon  existing  practices  to  employ  standardized  processes,  methods,  and  tools  in 
the  provided  common  tailorable  framework. 

Recommended  Follow-On  Work 

The  research  revealed  several  opportunities  for  future  work  that  was  not  within  the  scope 
of  the  research. 

1.  Further  customization  and  tailoring  of  the  framework,  to  translate/adapt  to  6.1 
projects,  to  include  6.1  specific  tools,  and  to  add  the  ability  to  tailor  at  the 
component  or  technology  level  within  the  “Integration  Level”  discriminant. 

2.  Incorporate  AFRL  specific  “tools  /  best  practices”  not  listed  in  the  SETT-STP 
framework.  The  tool  has  application  to  every  technology  directorate,  but  there 
may  be  additional  tailoring  (additions  or  subtractions)  needed  for  the  SE  activity 
list  in  SETT-STP.  An  example  would  be  adding  specific  activities  from  the 
Rational  Unified  Process  (RUP)  used  by  AFRL/RI. 
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3.  Validate  the  specific  language  within  the  SE  Taxonomy  activities  to  enable 
formal  AFRL  endorsement.  A  recommended  avenue  to  accomplish  this  is  to 
create  an  AFRL  Integrated  Dictionary  for  terms  within  the  SE  Taxonomy. 

4.  Determine  which  reference  materials  will  be  formally  accepted  within  AFRL 
framework,  as  the  INCOSE  Handbook  [INCOSE,  2006]  and  the  Buede  textbook 
[Buede,  1999]  were  utilized  for  the  framework. 

5.  Provide  specific  guidance  as  to  how  to  interpret  the  “SE  Rigor”  results  from  the 
framework.  The  research  provides  a  recommended  interpretation  as  a  starting 
point. 

6.  Implement  a  level  of  standardization  across  directorates  by  providing  instruction 
as  to  how  to  establish  the  context  of  the  project  being  evaluated,  whether  it  be  a 
specific  technology  project  or  an  FLTC  designator  at  the  Problem  (X.X)  or 
Capability  (X.X.X)  level.  The  SE  activities  should  be  applied  to  the  same  context 
(possibly  add  room  on  front  page  of  tool  to  identify  scope  or  the  context  of  the 
project). 

7.  Add  inputs,  outputs,  constraints,  sequencing,  and  related  activities  for  each  SE 
activity.  This  will  transform  the  SE  taxonomy  from  just  a  list  of  (sequential) 
activities  into  a  tool  that  incorporates  project  flow  and  emphasizes  the 
relationships  between  the  SE  activities. 
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Appendix  A.  Case  Study  Pre-Survey  Questionnaire 


Your  project  has  been  identified  for  inclusion  as  a  case  study  in  the  “Systems 
Engineering  in  a  Science  &  Technology  Environment”  thesis  project  at  the  Air  Force 
Institute  of  Technology.  The  following  initial  information  regarding  your  project  is 
requested  in  order  to  better  approach  interactions  during  the  case  study  period  of  the 
thesis  project: 

Project  Name: 

Point  of  Contact: 

AFRL  Directorate: 

Phone  Number: 

Email: 

Please  identify  where  your  project  falls  with  respect  to  the  following  areas.  Choose  a 
single  best  answer  if  possible.  Descriptions  are  provided  on  the  following  pages: 

1)  Primary  RDT&E  Budget  Category 

6.1  Basic  Research 

6.2  Applied  Research 

6.3  Advanced  Technology  Development 

2)  Total  S&T  Project  Budget 

Budget<  $500K 
$500K  <  Budget  <  $5M 
Budget  >  $5M 

3)  Core  Processes 

CP-1  CP-2  CP-3 

4)  Technology  Readiness  Level 

TRL1-2  TRL3-4  TRL  5-6  TRL  7-9 

5)  Level  of  Integration  /  System  Hierarchy 

Subsystem  Level  Technology 
System  Level  Concept 
Mission  Level  Concept 

6)  Requirements  Maturity 

Technology  Push 
Requirements  Pull 
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DISCRIMINANT  #1:  Research,  Development,  Test  and  Evaluation  (RDT&E) 
Budget  Category 

Source:  DoD  Financial  Management  Regulation  Volume  2B,  Chapter  5,  July  2008 
Reference: 

http://www.dtic.mil/descriptivesum/RDTE  Budget  Activies  Establishing  RDTE  Program  Elements.pdf 

DOMAIN  VALUE:  Budget  Activity  1,  Basic  Research.  Basic  research  is  systematic  study  directed 
toward  greater  knowledge  or  understanding  of  the  fundamental  aspects  of  phenomena  and  of  observable 
facts  without  specific  applications  towards  processes  or  products  in  mind.  It  includes  all  scientific  study 
and  experimentation  directed  toward  increasing  fundamental  knowledge  and  understanding  in  those  fields 
of  the  physical,  engineering,  environmental,  and  life  sciences  related  to  long-term  national  security  needs. 

It  is  farsighted  high  payoff  research  that  provides  the  basis  for  technological  progress.  Basic  research  may 
lead  to:  (a)  subsequent  applied  research  and  advanced  technology  developments  in  Defense-related 
technologies,  and  (b)  new  and  improved  military  functional  capabilities  in  areas  such  as  communications, 
detection,  tracking,  surveillance,  propulsion,  mobility,  guidance  and  control,  navigation,  energy  conversion, 
materials  and  structures,  and  personnel  support.  Program  elements  in  this  category  involve  pre-Milestone  A 
efforts. 

DOMAIN  VALUE:  Budget  Activity  2,  Applied  Research.  Applied  research  is  systematic  study  to 
understand  the  means  to  meet  a  recognized  and  specific  need.  It  is  a  systematic  expansion  and  application 
of  knowledge  to  develop  useful  materials,  devices,  and  systems  or  methods.  It  may  be  oriented,  ultimately, 
toward  the  design,  development,  and  improvement  of  prototypes  and  new  processes  to  meet  general 
mission  area  requirements.  Applied  research  may  translate  promising  basic  research  into  solutions  for 
broadly  defined  military  needs,  short  of  system  development.  This  type  of  effort  may  vary  from  systematic 
mission-directed  research  beyond  that  in  Budget  Activity  1  to  sophisticated  breadboard  hardware,  study, 
programming  and  planning  efforts  that  establish  the  initial  feasibility  and  practicality  of  proposed  solutions 
to  technological  challenges.  It  includes  studies,  investigations,  and  non-system  specific  technology  efforts. 
The  dominant  characteristic  is  that  applied  research  is  directed  toward  general  military  needs  with  a  view 
toward  developing  and  evaluating  the  feasibility  and  practicality  of  proposed  solutions  and  determining 
their  parameters.  Applied  Research  precedes  system  specific  technology  investigations  or  development. 
Program  control  of  the  Applied  Research  program  element  is  normally  exercised  by  general  level  of  effort. 
Program  elements  in  this  category  involve  pre-Milestone  B  efforts,  also  known  as  Concept  and  Technology 
Development  phase  tasks,  such  as  concept  exploration  efforts  and  paper  studies  of  alternative  concepts  for 
meeting  a  mission  need. 

DOMAIN  VALUE:  Budget  Activity  3,  Advanced  Technology  Development  (ATP).  This  budget  activity 
includes  development  of  subsystems  and  components  and  efforts  to  integrate  subsystems  and  components 
into  system  prototypes  for  field  experiments  and/or  tests  in  a  simulated  environment.  ATD  includes  concept 
and  technology  demonstrations  of  components  and  subsystems  or  system  models.  The  models  may  be 
form,  fit  and  function  prototypes  or  scaled  models  that  serve  the  same  demonstration  purpose.  The  results 
of  this  type  of  effort  are  proof  of  technological  feasibility  and  assessment  of  subsystem  and  component 
operability  and  producibility  rather  than  the  development  of  hardware  for  service  use.  Projects  in  this 
category  have  a  direct  relevance  to  identified  military  needs.  Advanced  Technology  Development 
demonstrates  the  general  military  utility  or  cost  reduction  potential  of  technology  when  applied  to  different 
types  of  military  equipment  or  techniques.  Program  elements  in  this  category  involve  pre-Milestone  B 
efforts,  such  as  system  concept  demonstration,  joint  and  Service-specific  experiments  or  Technology 
Demonstrations  and  generally  have  Technology  Readiness  Levels  of  4,  5,  or  6.  Projects  in  this  category  do 
not  necessarily  lead  to  subsequent  development  or  procurement  phases,  but  should  have  the  goal  of  moving 
out  of  Science  and  Technology  (S&T)  and  into  the  acquisition  process  within  the  future  years  defense 
program  (FYDP).  Upon  successful  completion  of  projects  that  have  military  utility,  the  technology  should 
be  available  for  transition. 
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DOMAIN  VALUE:  OTHER.  This  includes  Budget  Activity  4,  Advanced  Component  Development  and 
Prototypes  (ACD&P);  Budget  Activity  5,  System  Development  and  Demonstration  (SDD);  Budget  Activity 
6,  RDT&E  Management  Support;  and  Budget  Activity  7,  Operational  System  Development. 

Budget  Activity  4,  Advanced  Component  Development  and  Prototypes  (ACD&P).  Efforts  necessary  to 
evaluate  integrated  technologies,  representative  modes  or  prototype  systems  in  a  high  fidelity  and  realistic 
operating  environment  are  funded  in  this  budget  activity.  The  ACD&P  phase  includes  system  specific 
efforts  that  help  expedite  technology  transition  from  the  laboratory  to  operational  use.  Emphasis  is  on 
proving  component  and  subsystem  maturity  prior  to  integration  in  major  and  complex  systems  and  may 
involve  risk  reduction  initiatives.  Program  elements  in  this  category  involve  efforts  prior  to  Milestone  B 
and  are  referred  to  as  advanced  component  development  activities  and  include  technology  demonstrations. 
Completion  of  Technology  Readiness  Levels  6  and  7  should  be  achieved  for  major  programs.  Program 
control  is  exercised  at  the  program  and  project  level.  A  logical  progression  of  program  phases  and 
development  and/or  production  funding  must  be  evident  in  the  FYDP. 

Budget  Activity  5,  System  Development  and  Demonstration  (SDD).  SDD  programs  have  passed  Milestone 
B  approval  and  are  conducting  engineering  and  manufacturing  development  tasks  aimed  at  meeting 
validated  requirements  prior  to  full-rate  production.  This  budget  activity  is  characterized  by  major  line  item 
projects  and  program  control  is  exercised  by  review  of  individual  programs  and  projects.  Prototype 
performance  is  near  or  at  planned  operational  system  levels.  Characteristics  of  this  budget  activity  involve 
mature  system  development,  integration  and  demonstration  to  support  Milestone  C  decisions,  and 
conducting  live  fire  test  and  evaluation  (LFT&E)  and  initial  operational  test  and  evaluation  (IOT&E)  of 
production  representative  articles.  A  logical  progression  of  program  phases  and  development  and 
production  funding  must  be  evident  in  the  FYDP  consistent  with  the  Department’s  full  funding  policy. 

Budget  Activity  6,  RDT&E  Management  Support.  This  budget  activity  includes  research,  development,  test 
and  evaluation  efforts  and  funds  to  sustain  and/or  modernize  the  installations  or  operations  required  for 
general  research,  development,  test  and  evaluation.  Test  ranges,  military  construction,  maintenance  support 
of  laboratories,  operation  and  maintenance  of  test  aircraft  and  ships,  and  studies  and  analyses  in  support  of 
the  RDT&E  program  are  funded  in  this  budget  activity.  Costs  of  laboratory  personnel,  either  in-house  or 
contractor  operated,  would  be  assigned  to  appropriate  projects  or  as  a  line  item  in  the  Basic  Research, 
Applied  Research,  or  Advanced  Technology  Development  program  areas,  as  appropriate.  Military 
construction  costs  directly  related  to  major  development  programs  are  included. 

Budget  Activity  7,  Operational  System  Development.  This  budget  activity  includes  development  efforts  to 
upgrade  systems  that  have  been  fielded  or  have  received  approval  for  full  rate  production  and  anticipate 
production  funding  in  the  current  or  subsequent  fiscal  year.  All  items  are  major  line  item  projects  that 
appear  as  RDT&E  Costs  of  Weapon  System  Elements  in  other  programs.  Program  control  is  exercised  by 
review  of  individual  projects.  Programs  in  this  category  involve  systems  that  have  received  Milestone  C 
approval.  A  logical  progression  of  program  phases  and  development  and  production  funding  must  be 
evident  in  the  FYDP,  consistent  with  the  Department’s  full  funding  policy. 

050202  Establishing  RDT&E  Program  Elements 

A.  The  program  element  is  the  primary  data  element  in  the  Future  Years  Defense  Program  (FYDP)  and  is 
the  major  aggregation,  at  which  RDT&E  efforts  are  organized,  budgeted  and  reviewed.  All  funding 
associated  with  a  major  system  new  start  should  be  identified  in  a  unique  program  element.  Requests  to 
establish  program  elements  should  be  forwarded  to  OSD  Program  Analysis  and  Evaluation  for  coordination 
and  approval.  Instructions  are  contained  in  DoD  7045. 7-H,  “The  FYDP  Program  Structure  Handbook.” 

B.  In  general,  the  coding  symbology  identifies  the  RDT&E  budget  activity  for  the  program  element. 
Program  elements  in  RDT&E  budget  activities  1  through  6  will  have  “06”  in  the  first  two  positions; 
“06”indicates  it  is  part  of  Major  Force  Program  6,  Research  and  Development.  The  third  and  fourth 
position  will  identify  the  specific  budget  activity  (e.g.,  0602  is  an  RDT&E  budget  activity  2  program 
element).  Program  elements  in  RDT&E  budget  activity  7  reflect  the  Major  Program  of  the  fielded  system  in 
the  first  two  positions  (e.g.,  “01”  indicates  a  strategic  system). 
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DISCRIMINANT  #2:  Total  S&T  Project  Budget 

This  descriminant  captures  the  total  funding  size  of  the  S&T  project  across  fiscal  years.  As  it  is  used 
to  distinguish  relative  size  of  the  project,  the  division  of  domain  values  was  set  to  discriminate  very 
small  projects  from  very  large  projects  such  as  an  integrated  Advanced  Technology  Demonstration. 

“Budget”  refers  to  the  estimated  project  costs  for  the  lifetime  of  the  science  &  technology  program.  It 
includes  costs  for  the  prior,  current  and  planned  future  years  of  the  project’s  research  and/or  demonstration 
phase  (as  opposed  to  full  system  development  or  production.) 

Note:  If  the  project  is  an  integrated  demonstration,  do  NOT  include  costs  of  subsystem  or  component-level 
projects  being  done  under  separate  project  budget  authority. 

DOMAIN  VALUE:  Budget  <  $500K 

DOMAIN  VALUE:  $500K  <  Budget  <  $5M 

DOMAIN  VALUE:  Budget  >  $5M 


DISCRIMINANT  #3:  CoreProcesses 

DOMAIN  VALUE:  Core  Process  1  (CPI) 

Projects  that  address  future  technology  concepts  to  senior  Air  Force  leadership  and/or  advance  a  core 
technology  that  influences  the  broader  S&T  community  are  referred  to  CPI.  Projects  progressing  through 
CPI  that  have  identified  a  transition  customer  are  then  referred  to  CP2  to  continue  maturation  of  the 
technology,  ready  for  integration  into  an  acquisition  or  sustainment  program. 

Source:  AFRL  Enterprise  Process  Management  -  Volume  II:  Core  Process  2,  Paragraph 
4. 1.1. 1.1/2 

DOMAIN  VALUE:  Core  Process  2  (CP2) 

CP2  is  the  process  that  enables  AFRL  to  identify  and  mature  technologies  needed  to  enhance  or  transform 
weapons  systems  and  contribute  to  a  successful  technology  transition  process.  It  is  designed  to  have  strong 
ties  to  acquisition,  sustainment,  and  industrial  communities  and  to  focus  on  product  delivery  -  the  emphasis 
is  on  developing  and  delivering  affordable,  timely,  and  transitionable  technology  options  characterized  by 
disciplined  program  management  and  systems  engineering  and  heavily  drawing  upon  the  research  from 
Core  Process  1 .  Therefore,  the  primary  outputs  of  CP2  are  mature  technologies  ready  for  integration  into 
an  acquisition  or  sustainment  program  -  technologies  that  shape  today’s  Air  Force. 

Source:  AFRL  Enterprise  Process  Management  -  Volume  II:  Core  Process  2,  Paragraph 
1.3 

DOMAIN  VALUE:  Core  Process  3  (CP3) 

Core  Process  3  (CP3)  addresses  near-term  warfighter  technology  needs  through  the  rapid  infusion, 
integration,  and  innovation  of  S&T-based  solutions  that  capitalize  on  the  breadth  and  depth  of  AFRL’s 
expertise.  CP3  is  designed  to  tightly  integrate  AFRL  S&T  knowledge  with  operator  knowledge  to  deliver 
solutions  to  the  warfighter  in  6-12  months.  The  solutions  may  utilize  individual  or  focused  technology 
application  (in  which  case  the  process  is  usually  executed  at  the  TD  level),  or  cross-  and  multi-discipline 
technology  solutions  (executed  at  the  enterprise  level).  CP3  encompasses  technology  demonstrations  and 
corporate  efforts  for  consulting  and  prototyping  to  meet  near  term  warfighter  technology  needs.  CP3 
requires  a  framework  that  tolerates  risk-taking  and  innovative,  unconventional  (out-of-the-box)  thinking, 
yet  focuses  on  delivering  viable  solutions.  To  provide  these  attributes,  CP3  requires  the  cultural, 
institutional,  and  business  support  systems  needed  to  rapidly  deliver  innovative  capability  to  Air  Force  and 
other  AFRL  customers  and  stakeholders. 

Source:  AFRL  Enterprise  Process  Management  -  Volume  III:  Core  Process  3,  Paragraph 
1.3 
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DISCRIMINANT  #4:  Technology  Readiness  Level 

Technology  Readiness  Levels  are  determined  by  a  Technology  Readiness  Assessment  (TRA),  a  “regulatory 
information  requirement  for  all  acquisition  programs.  It  is  a  systematic,  metrics-based  process  that 
establishes  the  maturity  of  critical  technology  elements.  The  TRA  should  be  conducted  concurrently  with 
other  technical  reviews  such  as  the  Alternative  Systems  Review,  System  Requirements  Review,  or  the 
Production  Readiness  Review.  (Defense  Acquisition  Guidebook) 

Source:  Glossary  of  Defense  Acquisition  Acronyms  &  Terms,  12th  Edition,  July  2005. 
Reference:  https://akss.dau.mil/isp/glossarv.pdf 

DOMAIN  VALUE:  TRL  1-2 

TRL-1:  Basic  principles  observed  and  reported. 

Description:  Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be  translated  into  applied 
research  and  development.  Examples  might  include  paper  studies  of  a  technology's  basic  properties. 

TRL-2:  Technology  concept  and/or  application  formulated. 

Description:  Invention  begins.  Once  basic  principles  are  observed,  practical  applications  can  be  invented. 
Applications  are  speculative  and  there  may  be  no  proof  or  detailed  analysis  to  support  the  assumptions. 
Examples  are  limited  to  analytic  studies. 

DOMAIN  VALUE:  TRL  3-4 

TRL-3:  Analytical  and  experimental  critical  function  and/or  characteristic  proof  of  concept. 

Description:  Active  research  and  development  is  initiated.  This  includes  analytical  studies  and  laboratory 
studies  to  physically  validate  analytical  predictions  of  separate  elements  of  the  technology.  Examples 
include  components  that  are  not  yet  integrated  or  representative. 

TRL-4:  Component  and/or  breadboard  validation  in  laboratory  environment. 

Description:  Basic  technological  components  are  integrated  to  establish  that  they  will  work  together.  This 
is  relatively  "low  fidelity"  compared  to  the  eventual  system.  Examples  include  integration  of  "ad  hoc" 
hardware  in  the  laboratory. 

DOMAIN  VALUE:  TRL  5-6 

TRL-5:  Component  and/or  breadboard  validation  in  relevant  environment. 

Description:  Fidelity  of  breadboard  technology  increases  significantly.  The  basic  technological 
components  are  integrated  with  reasonably  realistic  supporting  elements  so  it  can  be  tested  in  a  simulated 
environment.  Examples  include  "high  fidelity"  laboratory  integration  of  components. 

TRL-6:  System/subsystem  model  or  prototype  demonstration  in  a  relevant  environment. 

Description:  Representative  model  or  prototype  system,  which  is  well  beyond  that  of  TRL  5,  is  tested  in  a 
relevant  environment.  Represents  a  major  step  up  in  a  technology's  demonstrated  readiness.  Examples 
include  testing  a  prototype  in  a  high-fidelity  laboratory  environment  or  in  simulated  operational 
environment. 

DOMAIN  VALUE:  TRL  7-9 

TRL-7:  System  prototype  demonstration  in  an  operational  environment. 

Description:  Prototype  near,  or  at,  planned  operational  system.  Represents  a  major  step  up  from  TRL  6, 
requiring  demonstration  of  an  actual  system  prototype  in  an  operational  environment  such  as  an  aircraft, 
vehicle,  or  space.  Examples  include  testing  the  prototype  in  a  test  bed  aircraft. 

TRL-8:  Actual  system  completed  and  qualified  through  test  and  demonstration. 

Description:  Technology  has  been  proven  to  work  in  its  final  form  and  under  expected  conditions.  In 
almost  all  cases,  this  TRL  represents  the  end  of  true  system  development.  Examples  include  developmental 
test  and  evaluation  of  the  system  in  its  intended  weapon  system  to  determine  if  it  meets  design 
specifications. 

TRL-9:  Actual  system  proven  through  successful  mission  operations. 

Description:  Actual  application  of  the  technology  in  its  final  form  and  under  mission  conditions,  such  as 
those  encountered  in  operational  test  and  evaluation.  Examples  include  using  the  system  under  operational 
mission  conditions. 
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DISCRIMINANT  #5:  Level  of  Integration  /  Demonstration  -OR-  System 
Hierarchy 

The  Air  Force  S&T  Vision  is  “Anticipate,  Find,  Fix,  Track,  Target,  Engage,  and  Assess  -  Anything, 
Anywhere,  Anytime”  (p.4).  In  order  to  realize  this  Vision,  the  Air  Force  Research  Laboratory  has  divided 
their  projects  into  Focused  Long  Term  Challenges  (FLTCs)  in  order  to  characterize  the  Air  Force  problem 
space  and  provide  a  framework  for  long  term  S&T  planning.  The  FLTC  framework  facilitates  a  dialog 
with  stakeholders  of  planning  priorities  and  desired  effects  without  prematurely  dictating  “solutions, 
platforms,  or  domain  specific  assumptions”  (p.8)  or  the  “type,  source,  or  timing  of  potential  technical 
solutions”(p.30).  FLTCs  are  currently  divided  into  the  following  categories:  Technology  Challenges, 
Problem  Statements,  Attributes,  Products,  and  Programs. 

Source:  AFRL  Capability  Based  S&T  Strategy  2030, 31  July  2007. 

In  order  to  make  the  taxonomy  more  generic  and  applicable  to  the  AFRL  context  as  well  as  other 
government  agency  (OGA)  and  industry  contexts,  the  following  domain  values  describe  the  level  of 
integration  and/or  demonstration  of  a  concept  or  technology  into  an  applicable  system  hierarchy: 

Source:  Student  Defined 

DOMAIN  VALUE:  Subsystem  Level  (or  below)  Technology 

Target  project/demonstration  is  at  the  subsystem  level  (or  lower  ...  i.e.  component).  A  fully  developed 
system  concept  may  not  yet  exist.  A  wide  range  of  external  dependencies  are  possible,  and  may  be  only 
notionally  defined.  It  is  also  possible  that  a  target  system/component  is  already  identified. 

DOMAIN  VALUE:  System  Level  Concept  /  Demonstration 

Target  project/demonstration  is  contained  to  a  specific  system  and/or  S&T  project.  Will  generally  integrate 
subsystem  and/or  component  technologies  within  the  system  concept.  Interfaces  are  well  understood  and 
within  control  of  the  project  lead. 

DOMAIN  VALUE:  Mission  Level  Concept  /  Demonstration 

Target  project/demonstration  includes  multiple  independent  systems  and/or  project  interfaces  and  may 
require  integration  at  levels  beyond  the  control  of  the  project  lead,  and  will  generally  have  dependencies 
external  to  the  project. 

DISCRIMINANT  #6:  Requirements  Maturity 

Source:  AF  63-101  “Operations  of  Capability  Based  Acquisition  Systems” 

DOMAIN  VALUE:  Technology  Push 

Technology  push  is  defined  as  technology  that  has  the  potential  for  new  revolutionary  warfighting 
capabilities  (AF  63-101,  2.1.3). 

DOMAIN  VALUE:  Requirements  Pull 

Requirements  pull  is  defined  as  technology  developed  in  response  to  documented  operator  needs  (AF  63- 
101,2.1.3). 
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Appendix  B.  SE  Tailoring  Tool  User’s  Guide 


Systems  Engineering  Tailoring  Tool 
for  S&T  Projects 

User’s  Guide  (Version  1.0,  4  March  2009) 
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Welcome  to  the  Systems  Engineering  Tailoring  tool  for  Science  &  Technology  Projects  (SETT-STP) 


Please  select  the  following  project  discriminants  that  apply.  Hover  on  a  selection  to  see  a  description. 

(To  select  a  discriminant  place  a  "1"  in  the  cell  next  to  the  domain  value.  Leave  all  other  domain  values  marked  with 
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After  selecting  your  project  discriminant  click  on  the  worksheet  tab  "Tailored  SE  Activities"  below. 
Drill  down  into  lower  level  SE  activities  by  using  the  "+"  marks  on  the  left  of  that  worksheet. 
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Introduction 


Systems  engineering  (SE)  is  essential  to  effectively  developing  and  transitioning  complex 
technical  products,  whether  they  transition  to  a  higher  Technology  Readiness  Level,  to  a 
more  mature  Science  and  Technology  (S&T)  effort,  to  a  Systems  Program  Office,  or  to 
an  operational  user.  However,  executing  the  complete  set  of  SE  activities  can  be 
cumbersome,  confusing,  and  not  resource  effective  for  the  laboratory  scientist. 

This  SE  Tailoring  Tool  was  developed  to  help  S&T  project  managers  decide  which  SE 
activities  are  either  critical  or  of  minimal  importance  for  a  certain  type  of  project.  It 
recognizes  that  all  projects  are  not  the  same,  and  the  maturity  of  the  technology,  the 
development  lead-time,  and  even  the  project  budget  dictate  the  SE  rigor  that  should  be 
applied. 

The  tool  provides  a  relative  importance  “weight”  for  over  350  individual  SE  activities 
based  on  the  type  of  project.  The  results  do  not  provide  a  “yes/no”  checklist  as  to  what  to 
do  and  what  not  to  do,  but  rather  suggests  a  relative  importance  of  activities  to 
accomplish. 

A  few  Notes  before  getting  started: 

The  tool  is  a  starting  point;  not  all  projects  will  clearly  fit  the  tailoring  for  the 
selected  categories.  Users  may  need  to  adjust  their  selections  to  adjoining 
domain  values  to  get  the  tailoring  that  best  suits  their  specific  program. 

The  tool  describes  WHAT  to  do,  not  necessarily  HOW  to  do  it.  It  assumes  the 
user  is  familiar  with  basic  SE  terminology  and  principles  and  has  access  to  more 
detailed  information  about  specific  tasks  and  suggested  tools.  The  tool  does 
provide  ready  references  to  defense,  industry,  and  academic  sources  to  aid  the 
user  on  HOW  to  implement  the  activities. 

The  tailoring  tool  can  be  an  effective  teaching  aid,  but  it  is  not  designed  to  be  a 
self-contained  SE  training  tool. 

Changes  to  the  weighting  can  be  made  if  one  discriminant  improperly  dominates 
the  weighting  for  a  given  project.  See  the  “Advanced  User  Notes”  section  to 
adjust  the  weighting  factors. 

Although  users  will  be  asked  to  select  specific  domain  values  for  their  project, 
the  best  results  will  be  achieved  if  the  user  has  a  clearly  defined  context  for  the 
project  as  a  sanity  check  on  the  results.  For  example,  is  the  program  a  portion  of 
a  larger  project  effort,  and  what  is  the  transition  target  for  the  program  or 
project.  Without  the  proper  project  context,  the  recommended  SE  activities  will 
not  provide  the  true  systems  engineering  value  to  the  project.  All  domain  values 
should  be  based  on  the  discriminant’s  desired  end-state  for  the  current  phase  of 
the  project. 
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Getting  Started 


To  start  using  the  tool,  open  the  file:  SE  Tailoring  Framework  for  S&T  Projects  vl  O.xls 
on  a  computer  equipped  with  Excel  2007.  Older  versions  of  Excel  will  still 
fundamentally  run  the  tool,  but  may  lose  some  advanced  functionality.  The  tool  opens  to 
the  worksheet  titled  “Tailoring  Section”  and  contains  six  blocks  that  describe  the 
discriminants  and  domain  values  (see  Figure  1).  Each  domain  value  has  a  definition 
embedded  in  the  cell  comment,  so  hovering  over  the  red  triangle  in  the  upper  right  corner 
of  the  cell,  or  right  clicking  on  “Show  Comment”  for  a  cell  will  display  the  definition. 

For  a  given  project,  the  user  should  type  a  “1”  in  the  appropriate  block  for  each  domain 
value  that  applies,  and  type  a  “0”  for  each  value  that  does  not  apply.  An  optional  space 
allows  for  saving  the  Project  Name  and  Point  of  Contact. 
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Figure  10:  Tailoring  Selection  Screen 

Note:  The  base  file  is  “read  only”,  so  users  will  need  to  save  the  tailoring  tool  file  for 
each  set  of  project  discriminants.  This  will  ensure  a  common  starting  point  for  all 
projects,  consistent  display  of  results,  and  will  maintain  an  unaltered  version  of  the  tool 
for  later  user. 
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Displaying  Your  Results 


After  the  appropriate  domain  values  are  selected,  click  on  the  tab  to  select  the  “Tailored 
SE  Activities”  Worksheet  (see  Figure  2).  The  “Tailored  SE  Activities”  screen  contains 
the  Defense  Acquisition  Guidebook’s  Technical  Processes  (TPs)  and  Technical 
Management  Processes  (TMPs),  the  activity  detail  level,  the  SE  activity  name,  the 
literature  source  that  the  activity  was  derived  from,  the  tools  associated  with  the  activity, 
and  the  tailored  weight  of  that  activity. 


The  tailoring  weights  are  only  directly  applied  to  activities  at  level  3.  The  level  3  scores 
are  rolled  up  for  level  2  weights,  which  are  represented  by  the  highest  and  lowest  activity 
weights  from  the  subordinate  level  3  activities.  Likewise,  the  level  1  weights  display  the 
maximum  and  minimum  weights  from  each  of  the  level  2  categories.  The  scores  will  be 
in  a  range  of  0%  to  100%,  with  a  100%  activity  being  critical  to  successful  SE  on  the 
project,  and  0%  being  of  minimal  impact.  The  default/initial  display  will  show  the  roll¬ 
up  weights  associated  with  levels  1  and  2. 
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Note:  The  tailoring  weights  for  SE  Activities  will  vary  if  something  other  than  one 
domain  value  is  selected  for  each  project  discriminant.  For  instance,  if  two  values  for 
Technology  Readiness  Level  are  selected,  the  weights  will  be  different  than  if  only  one 
TRL  value  is  selected.  Likewise,  if  one  discriminant  is  left  with  all  “0”  values  selected, 
the  weights  will  be  different  than  if  one  domain  value  from  each  discriminant  was 
selected.  The  tailored  values  will  all  still  be  based  on  a  0-100%  weighting  scale. 

Grouping  and  Filtering 

There  is  a  “+”  sign  to  the  left  of  each  activity.  The  “+”  sign  can  be  clicked  for  each 
process  to  drill  down  to  the  next  level  of  SE  activity  grouping.  This  process  can  be 
repeated  to  expand  activity  details  up  to  five  levels,  depending  on  the  specific  details 
included  in  the  activity.  Alternately,  activity  details  can  be  expanded  by  clicking  on  the 
desired  detail  level  (1,  2,  3,  4,  or  5)  from  the  upper  left  corner  of  the  worksheet. 

At  the  top  of  each  data  column,  a  dropdown  box  with  a  downward  pointing  arrow  is 
displayed.  These  arrows  allow  for  filtering  on  a  specific  piece  of  data  within  each 
column.  For  instance,  if  a  user  only  wanted  to  display  activities  in  TMP-5,  the  user 
would  click  on  the  dropdown  box  in  column  A  “Process”,  then  uncheck  all  boxes  except 
for  TMP-5. 

Note:  Users  are  encouraged  to  use  the  filtering  for  the  “Process”,  “Level”,  and  “Tailored 
Weight”  data  columns  for  displaying  data.  Use  filtering  for  the  “SE  Activity”,  “Source”, 
and  “Tools”  data  columns  only  to  find  specific  items,  or  use  the  search  feature  in  Excel. 

Note:  The  filtering  will  override  the  previously  selected  display  grouping.  The  grouping 
can  be  restored  for  the  filtered  selection  by  clicking  on  the  appropriate  “+”  or 
Clicking  on  the  grouping  number  boxes  in  the  upper  left  comer  will  override  the  filtering 
selection. 


82 


Advanced  User  Notes 


The  SE  activity  weighting  factor  system  is  embedded  and  hidden  in  the  tool.  The  base 
factors  are: 

RDT&E  Category:  0.2 
Project  Budget:  0.3 
Core  Process:  0.2 
Technology  Readiness  Level:  0.2 
Integration  Level:  0.05 
Requirements  Maturity:  0.05 

The  factors  can  be  changed,  but  it  is  necessary  that  the  factors  sum  to  1.0  in  order  to  keep 
the  weighting  scale  at  0-100%.  To  access  them,  select  columns  “Y”  through  “AH”,  right 
click,  and  select  “unhide”.  Make  sure  that  all  rows  are  displayed;  this  is  best  done  by 
clicking  on  the  “5”  in  the  detail  level  selector  in  the  upper  left  corner  of  the  worksheet. 
The  factors  are  located  in  column  AE.  Hide  the  factors  by  highlighting  columns  “Z”, 
through  “AG”,  right  click,  and  select  “hide”.  DO  NOT  change  the  other  values  in 
columns  “Z”  through  “AF”  or  the  tailoring  weight  calculations  will  be  lost. 

Activities  can  be  added  or  deleted  by  inserting  or  deleting  a  selected  row,  respectively, 
within  Excel.  If  an  activity  is  added  at  Level  3,  tailoring  should  be  included  for  each 
possible  domain  value.  To  access  the  tailoring  markings,  select  columns  “E”  through 
“Y”,  right  click,  and  select  “unhide”.  The  discriminant  categories  and  domain  values  are 
listed  at  the  top  of  the  spreadsheet.  Place  a  “1”  in  the  appropriate  box  if  the  activity  is 
deemed  necessary  for  successful  completion  of  a  project  in  that  domain  value;  place  a  “0” 
in  that  box  otherwise.  To  display  the  calculated  weight,  select  the  cell  for  weight  of  an 
adjoining  level  3  activity,  copy  it,  and  paste  it  in  the  weight  column  of  the  new  activity. 
Do  NOT  drag  an  adjoining  weight  into  the  new  cell.  Hide  the  tailoring  values  by 
highlighting  columns  “F”,  through  “X”,  right  click,  and  select  “hide”.  After  inserting  or 
deleting  an  activity,  that  process’  activities  need  to  be  re-grouped.  Use  the  “group”  and 
“ungroup”  features  on  the  “data”  menu  in  Excel  2007. 
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Appendix  C.  Tailored  Weight  Statistical  Analysis 


The  tailored  weights  were  analyzed  to  understand  the  impact  of  each  discriminant  on  the 
total  tailored  weight  for  each  Level  3  activity.  The  baseline  project  type  for  the  analysis 
is  6.2,  $500K  -  $2M,  CP-2,  TRL  3-4,  System  Level,  and  Requirements  Pull.  From  this 
baseline,  each  discriminant  was  modified  to  each  contained  domain  value.  For  each 
domain  value,  the  tailored  weight  was  recorded  for  the  350  Level  3  activities.  The 
tailored  weights  were  then  binned,  with  bin  widths  of  10%,  to  output  total  numbers  of 
Level  3  activities  that  fall  in  each  bin.  It  is  important  to  note  that  weight  factors  were 
applied  as  follows:  RDT&E  Category:  0.2;  Project  Budget:  0.3;  Core  Process:  0.2;  TRL: 
0.2;  Integration  Level:  0.05;  and  Requirements  Maturity:  0.05.  The  data  and  analysis  for 
each  varied  discriminant  follows. 


Various  RDT&E  Categories,  $500K  -  $2M,  CP-2,  TRL  3-4,  System  Level,  Rqmts  Pull 


Bins 

6.1 

6.2 

6.3 

0-5% 

12 

12 

12 

5-15% 

10 

10 

4 

15-25% 

0 

0 

0 

25-35% 

24 

11 

0 

35-45% 

5 

3 

9 

45-55% 

14 

17 

0 

55-65% 

41 

12 

23 

65-75% 

2 

14 

0 

75-85% 

75 

43 

60 

85-95% 

0 

0 

0 

95-100% 

165 

228 

242 

Analysis:  This  discriminant  has  a  significant  effect  on  the  tailoring.  The  portions  of 
100%  weights  comprise  47%  of  all  6.1  activities,  65%  of  all  6.2  activities,  and  69%  of  all 
6.3  activities. 
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Various  Budget,  6.2,  CP-2,  TRL  3-4,  System  Level,  Rqmts  Pull 


Bins 

<$500K 

$500K- 

$2M 

>$2M 

0-5% 

12 

12 

12 

5-15% 

13 

10 

4 

15-25% 

0 

0 

0 

25-35% 

23 

11 

0 

35-45% 

0 

3 

9 

45-55% 

47 

17 

0 

55-65% 

0 

12 

23 

65-75% 

57 

14 

0 

75-85% 

13 

43 

60 

85-95% 

0 

0 

0 

95-100% 

185 

228 

242 

Analysis:  This  discriminant  has  a  significant  effect  on  the  tailoring.  The  portions  of 
100%  weights  comprise  53%  of  all  low  budget  activities,  65%  of  all  middle  budget 
activities,  and  69%  of  all  high  budget  activities.  Additionally,  the  portions  of  weights 
below  55%  comprise  27%  of  all  <$500K  activities,  15%  of  all  $500K-$2M  activities,  and 
only  7%  of  >$2M  activities. 


Various  Core  Process,  6.2,  $500K-$2M,  TRL  3-4,  System  Level,  Rqmts  Pull 


Tailoring  for  Various  Core  Processes,  6.2, 

$500K  -  $2M,  TRL  3-4,  System  Level,  Requirements  Pull 


Bins 

CP-1 

CP-2 

CP-3 

0-5% 

12 

12 

1 

5-15% 

21 

10 

0 

15-25% 

0 

0 

0 

25-35% 

17 

11 

21 

35-45% 

11 

3 

0 

45-55% 

10 

17 

17 

55-65% 

30 

12 

12 

65-75% 

4 

14 

14 

75-85% 

51 

43 

45 

85-95% 

0 

0 

0 

95-100% 

194 

228 

228 
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Analysis:  The  Core  Process  discriminant  affects  some  tailoring,  but  to  a  lesser  extent 
than  does  RDT&E  Category  and  Project  Budget.  The  portion  of  100%  weighted 
activities  is  55%  for  CP-1,  and  65%  for  both  CP-2  and  CP-3.  The  tailoring  stands  out  a 
bit  more  in  the  lower  weighted  regions,  as  the  portions  of  activities  with  weights  below 
55%  are  20%  of  CP-1,  15%  of  CP-2,  and  11%  of  CP-3. 


Various  Technology  Readiness  Levels,  6.2,  $500K-$2M,  CP-2,  System,  Rqmts  Pull 


Bins 

1-2 

3-4 

5-6 

7-9 

0-5% 

12 

12 

12 

12 

5-15% 

10 

10 

7 

1 

15-25% 

0 

0 

11 

0 

25-35% 

15 

11 

3 

9 

35-45% 

4 

3 

0 

0 

45-55% 

21 

17 

15 

15 

55-65% 

14 

11 

5 

4 

65-75% 

6 

14 

27 

27 

75-85% 

102 

43 

18 

20 

85-95% 

0 

0 

0 

0 

95-100% 

165 

228 

262 

262 

Analysis:  The  TRL  discriminant  has  a  moderate  effect  on  tailoring,  especially  at  the 
lower  domain  values.  The  portions  of  100%  weights  are  47%  for  TRL  1-2,  65%  for  TRL 
3-4,  and  75%  for  TRL  5-9.  Additionally,  the  lower  weights  are  reflective  of  this 
moderate  effect  on  tailoring.  The  portions  of  activities  weighted  below  55%  are  18%  for 
TRL  1-2,  15%  for  TRL  3-4,  14%  for  TRL  5-6,  and  11%  for  TRL  7-9. 
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Various  Integration  Levels,  6.2,  $500K-$2M,  CP-2,  TRL  3-4,  Rqmts  Pull 


Tailoring  for  Various  Integration  Levels,  6.2,  $500K  - 
$2M,  CP-2,  TRL  3-4,  Requirements  Pull 


Activity  Tailored  Weights 


■  Subsystem 

■  System 
i  Mission 


Bins 

Subsystem 

System 

Mission 

0-5% 

12 

12 

12 

5-15% 

10 

10 

10 

15-25% 

0 

0 

0 

25-35% 

11 

11 

11 

35-45% 

3 

3 

3 

45-55% 

17 

17 

17 

55-65% 

12 

12 

12 

65-75% 

14 

14 

14 

75-85% 

43 

43 

43 

85-95% 

0 

0 

0 

95-100% 

228 

228 

228 

Analysis:  Integration  Levels  had  no  effect  on  the  SE  activity  tailoring.  All  activities  fell 
into  the  same  weight  bins  for  each  integration  level  (mission,  system,  and  subsystem). 
This  supports  the  argument  to  apply  a  lower  weight  factor  to  this  discriminant. 

Various  Requirement  Maturity  Levels,  6.2,  $500K-$2M,  CP-2,  TRL  3-4,  System 


Bins 

Tech 

Push 

Rqmts 

Pull 

0-5% 

12 

12 

5-15% 

10 

10 

15-25% 

0 

0 

25-35% 

11 

11 

35-45% 

3 

3 

45-55% 

17 

17 

55-65% 

12 

12 

65-75% 

14 

14 

75-85% 

43 

43 

85-95% 

0 

0 

95-100% 

228 

228 

Analysis:  Requirement  Maturity  Levels  had  no  effect  on  the  SE  activity  tailoring.  All 
activities  fell  into  the  same  weight  bins  for  each  requirement  maturity  level  (Tech  Push 
and  Requirements  Pull).  This  supports  the  argument  to  apply  a  lower  weight  factor  to 
this  discriminant. 
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Appendix  D.  Glossary 


CASE  STUDY 

Current  and  recently  completed  S&T  project  reviewed  by  the  team  in  order  to  refine  the 
SE  taxonomy’s  terminology,  grouping,  or  tailoring  values. 

DISCRIMINANTS 

Six  categories  utilized  to  identify  various  aspects  of  a  project,  to  include:  Primary 
RDT&E  Budget  Category,  Total  S&T  Project  Budget,  Core  Process,  Technology 
Readiness  Level,  Level  of  Integration  /  System  Hierarchy,  and  Requirements  Maturity. 

DOMAIN  VALUES 

Eighteen  sub-categories  of  the  discriminants,  utilized  to  specify  project  information. 

FILTERING 

Tailoring  tool  feature  to  retrieve  specific  data  within  each  column  of  the  worksheet,  as 
indicated  by  “down  arrows”  in  the  column  title  blocks. 

FRAMEWORK 

Comprised  of  a  taxonomy  of  comprehensive  SE  activities  and  a  separate  taxonomy  of 
relevant  categories  and  domain  values  possible  for  S&T  projects,  and  forming  the  basis 
for  the  tailoring  tool. 

GROUPING 

Tailoring  tool  feature  to  display  the  SE  activity  details,  up  to  five  levels. 

MATURITY 

State  of  readiness  to  transition  to  the  next  level  of  development,  implementation,  or 
utilization. 

PROJECT 

Any  planned  effort  with  a  specific  end  goal. 

PROJECT  TAXONOMY 

Classifying  a  project  by  the  six  discriminants. 

QUALIFICATION 

The  process  of  verifying  and  validating  the  system  design  and  then  obtaining  the 
stakeholder's  acceptance  of  the  design,  per  Dennis  Buede. 

SE  ACTIVITIES 

Actions  done  in  the  nature  of  SE;  and  as  a  total  form  the  SE  taxonomy. 
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SE  PROCESSES 

A  structured  SE  activity  that  may  be  accompanied  by  specific  methods  and/or  tools.  Not 
to  be  confused  with  the  16  DAG  Technical  and  Technical  Management  Processes. 

SE  RIGOR 

The  amount  of  formal  planning,  coordination  and  documentation  applied  to  a  systems 
engineering  activity. 

SE  TAXONOMY 

Incorporate  a  broad  set  of  recognized  SE  activities  from  academic,  defense,  and  industry 
sources  and  organize  these  activities  according  to  the  Defense  Acquisition  Guidebook’s 
structure  of  Technical  Processes  (TPs)  and  Technical  Management  Processes  (TMPs). 

SE  TOOLS 

Means  to  accomplish  an  SE  activity. 

STAKEHOLDERS 

Persons  with  an  invested  interest  in  a  situation,  action  or  enterprise. 

STATE 

Set  of  designated  discriminant  domain  values  for  a  project. 

SUPER  SET 

Compilation  of  SE  activities  from  multiple  literature  sources  to  include:  academic, 
defense,  and  industry  perspectives. 

TAILOR 

Ability  to  select  SE  activities,  based  upon  the  domain  values  selected. 

TAILORED  WEIGHT 

For  a  given  set  of  SE  activities,  the  scores  were  summed  and  normalized  on  a  0-100% 
scale,  revealing  the  relative  importance  of  each  activity  for  a  specific  project  state,  to  give 
a  ranking  on  where  resources  should  be  applied. 

TIME  HORIZON 

Factors  include  the  end-state  TRL  level  and  total  project  budget,  as  opposed  to  the 
current  state  of  the  project.  Critical  to  establishing  the  context  of  the  project  to  which  SE 
activities  will  be  applied. 

TOOL  VALIDATION 

Providing  the  tool  to  potential  users  for  their  evaluation,  resulting  in  tool  refinement  for 
actual  utilization. 

TRANSITION 

Movement  of  the  project  to  the  next  level  of  development,  implementation,  or  utilization. 
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WEIGHT  FACTORS 

Importance  of  the  six  discriminants,  in  relation  to  each  other,  as  stated  in  percentages  to 
equal  100%. 


WORKBOOK 

The  total  contents  of  a  single  Excel  file.  A  workbook  may  consist  of  one  or  multiple 
worksheets. 

WORKSHEET 

Individual  tabs  within  the  tailoring  tool  (an  Excel  file)  to  include  Interface  and  Tailoring 
Results. 
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Appendix  E.  Acronyms 


ACD&P 

ACTD 

AFI 

AFIT 

AFMC 

AFMCI 

AFOSR 

AFRL 

AFRL/RX 

AFRL/XP 

AFRFI 

ATD 

CBP 

CP 

D&SWS 

DAG 

DoD 

EMRF 

FETC 

HEFETP 

ICOM 

IG 

INCOSE 

IPPD 

IRT 

LMR 

MRL 

PLOCAAS 
Pre-MS  A 

R&D 

RDT&E 


Advanced  Component  Development  and  Prototypes 

Advanced  Concept  Technology  Demonstrator 

Air  Force  Instruction 

Air  Force  Institute  of  Technology 

Air  Force  Materiel  Command 

Air  Force  Materiel  Command  Instruction 

Air  Force  Office  of  Scientific  Research 

Air  Force  Research  Laboratory 

Air  Force  Research  Laboratory  /  Materials  and  Manufacturing  Directorate 
Air  Force  Research  Laboratory  /  Plans  and  Programs 
Air  Force  Research  Laboratory  Instruction 
Advanced  Technology  Development 

Capability  Based  Planning 
Core  Process 

Developing  and  Sustaining  Weapons  Systems 
Defense  Acquisition  Guidebook 
Department  of  Defense 

Engineering  and  Manufacturing  Readiness  Level 

Focused  Long-Term  Challenge 

High  Energy  Laser  on  a  Large  Tactical  Platform 

Input,  Control,  Output,  Mechanism 
Inspector  General 

International  Council  on  Systems  Engineering 
Integrated  Process  and  Project  Development 
Independent  Review  Team 

Laboratory  Management  Review 

Manufacturing  Readiness  Level 

Powered  Low  Cost  Autonomous  Attack  System 
Commission  on  Pre-Milestone  A  Systems  Engineering  Report 

Research  and  Development 

Research  and  Development  Test  &  Equipment 
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S&T  Science  and  Technology 

SDD  System  Development  and  Demonstration 

SE  Systems  Engineering 

SEAM  System  Engineering  Assessment  Model 

SEC  Systems  Engineering  Council 

TASE  Transformational  Activities  in  Systems  Engineering  Report 

TMP  Technical  Management  Process 

TP  Technical  Process 

TPMM  Technology  Program  Management  Model 

TRL  Technology  Readiness  Level 
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Appendix  F.  Example  SETT-STP  Input  &  Output 


d  “1  '  -i  *  SE  Tailoring  Tool  for  Science  &  Technology  Projects  v1_0jcIs  [Read-Only]  [Compatibility  Mode]  -  Microsoft  Excel 

Home  Insert  Page  Layout  Formulas  Data  Review  View  Developer  Acrobat 


a  x 

a  X 


e  * 

Paste  j 
Clipboard  ft 


Tahoma 

B  I  U 


/.l 


Alignment 


Conditional  Format  Cell 
Formatting  •  as  Table  *  Styles 
Styles 


Insert  Delete  Format 
Cells 


&■ 

a- 


Sort  &  Find  & 
Filter  -  Select  * 


Welcome  to  the  Systems  Engineering  Tailoring  tool  for  Science  &  Technology  Projects  (SETT-STP) 


Latest  Change:  3/4/2009  9:25 


Please  select  the  following  project  discriminants  that  apply.  Hover  on  a  selection  to  see  a  description. 

(To  select  a  discriminant  place  a  "1”  in  the  cell  next  to  the  domain  value.  Leave  all  other  domain  values  marked  with  a  "0") 


RDT&E  Category 

6.1  (Basic  Research) 

3 

6.2  (Applied  Research) 

6.3  (Advanced  Technology  Development) 

Project  Budget 

12 

<$500K 

0 

13 

$500K  -  $2M 

1 

14 

>$2M 

0 

Technology  Readiness  Level 

' 

a 

3-4 

5-6 

7-9 

Integration  Level 

Subsystem 

System 

zM 

Mission 

° 

Core  Process  | 

CP-1  (Far  Term)  1  0  1 

Requirements  Maturity 

CP-2  (Medium  Term)  | 

Technology  Push  |  1 

CP-3  (Urgent  User  Needs) 

Requirements  Pull  ]\  0 

After  selecting  your  project  discriminant  click  on  the  worksheet  tab  "Tailored  SE  Activities"  below. 
Drill  down  into  lower  level  SE  activities  by  using  the  "+"  marks  on  the  left  of  that  worksheet. 


Systems  Engineering  Processes 


w 

iy 


es 

I  \ 


:al  Management  Proc*tMl(Contlno 


0  4&FIT 


PROJECT  NAME 


I  Requirements  pull  is  defined  as  technology  developed  m 
^response  to  documented  operator  needs  (AF  63-101,  2. 1.3). 


POINT  OF  CONTACT 


29  Major  Steve  Behm,  Major  Brad  Pitzer,  Ms.  Jane  White,  Dr.  David  Jacques  (faculty  advisor) 


33 

34 

H  i  I 


Tailoring  Selection  Tailored  SE  Activities  Tool  Notes  Glossary  Tab  °J 


JE 


Cell  E19  commented  by  Maj  Brad  Pitzer 


na 
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■flL« 


Process 

Leve 

SE  Activity 

Source 

Tools 

SE  Rigor 

TP-1 

1 

TP-1  (Requirements  Development! 

30%  100% 

TP  1 

2 

Establish  Communications  with  Stakeholders 

DAG  Sec  4.2.41 

30%  100% 

TP-1 

3 

Identify  stakeholders 

DAG  Sec  4.2.41.; 

Buede  Pg  23.  MCOSt 

Pg  4-3,  4.4 

100% 

TP-1 

3 

Address  customer  needs 

OAG  Sec  4.2.4. 1.; 
INCOSE  Pg4.2 

10 0% 

TP  1 

4 

Identify  stakeholder  requirements 

INCOSE  Pg  3.7 

TP  1 

A 

Oanfy  stakeholder  requirements 

INCOSE  Pg  3.7 

TP  1 

A 

Document  stakeholder  requirements 

INCOSE  Pg  3.7 

TP  1 

A 

Negotiate  modifications  to  resolve  inrealirab*  requirements 

INCOSE  Pg  4.4 

TP  1 

Avoid  acceptance  of  unrealistic  objectives 

INCOSE  Pg  4.4 

TP-1 

s 

Avoid  acceptance  of  competing  objectives 

INCOSE  Pg  4.4 

TP-1 

3 

Develop  stakeholder  agreements 

INCOSE  Pg  4.2  4 

Memorandum  of 
Agreement  / 

Understanding 

100% 

TP  1 

A 

Establish  good  relationship  with  stakeholders 

INCOSE  Pg  4.4 

TP  1 

A 

UUlue  terms  and  conditions  of  agreements 

INCOSE  Pg  4.2 

TP  1 

3 

Obtain  stakehoidc'  approval 

INCOSE  Pg4.3 

30% 

TP  1 

3 

Identify  transit**!  partners  and  relevant  applications 

(Student  De-'ved) 

100% 

TP-1 

2 

Identify  Project  Constraints 

DAG  Sec  4.2.4  1 

SO%  100% 

TP  1 

3 

Identify  constraints  from  agreements 

INCOSE  Pg  4.3 

100% 

TP  1 

3 

Identify  schedule  constraints 

DAG  Sec  4.2.4.I.; 

Buede  Pg  22,  MCOSE 
Pg4-2 

ntegrated  Master 
Schedule 

100% 

TP  1 

4 

Identify  schedule  issues 

Buede  Pg  4S 

TP  1 

4 

Identify  development  time  period 

Buede  Pg  4S.  131 

TP  1 

4 

Oetermme  manufacturing  tme  for  each  imrt 

Buede  Pg  131 

TP  1 

4 

Determine  training  time  to  reach  proficiency  by  category  of  user 

Buede  Pg  131 

TP  1 

4 

Determine  deployment  period 

Buede  Pg  131 

TP  1 

4 

Determine  durability  of  system 

Buede  Pg  131 

TP-1 

3 

Identify  cost  constraints 

DAG  Sec  4.2.4.I.; 

Buede  Pg  22.  4$; 
INCOSE  Pg  4.2 

Cost  Est  mates 

100% 

TP  1 

4 

Identify  cost  trade  offs 

Buede  Pg  4S 

TP  1 

4 

Determine  affordability 

Buede  Pg  131 

TP  1 

4 

Determine  development  cost 

Buede  Pg  131 

TP-1 

4 

Determine  production  cost  of  system 

Buede  Pg  131 

TP  1 

4 

Determine  decommissioning  cost  of  system 

Buede  Pg  131 

TP-1 

3 

Identify  technical  constraints 

OAG  Sec  4.2.4.I.; 

Buede  Pg  22,  MOOSE 
Pg4-2 

Technology  Readiness 
Assessment 

100% 

TP  1 

4 

Address  market  research 

DAG  Sec  4.2.4  1. 

TP  1 

4 

Identify  technology  to  be  incorporated 

Buede  Pg  4S 

TP  1 

3 

Identify  cost  performance  trade  offs 

Buede  Pg  4S 

100% 

TP  1 

3 

Identify  external  interface  constraints 

Buede  Pg  4S 

lOEKJ,  DoOAf  SV  1 

so% 

TP  1 

2 

Determine  Required  Capabilities 

DAG  Sec  4.2.4  1 

30%  100% 

TP-1 

3 

Identify  operations  needs 

OAG  Sec  4.2.4. 1. 

Capabilities  Review  & 
Risk  Assessment 
(CRRAI.  CONORS.  Use 
Cases 

60% 

TP  1 

4 

Identify  derived  requrements 

Buede  Pg  128 

TP  1 

4 

Identify  implied  retirements 

Buede  Pg  128 

TP  1 

4 

Identify  emergent  requirements 

Buede  Pg  128 

TP  1 

4 

Identify  operational  life 

Buede  Pg  4S 

TP-1 

4 

Determine  plan  for  disposal 

INCOSE  Pg  3.9,  4.3 

TP-1 

S 

Utilae  scenarios  to  define  dsposal  concept  documents 

INCOSE  Pg  4.4 

TP  1 

3 

Identify  capability  gaps 

DAG  Sec  4.2.4  1. 

60% 

TP  1 

3 

Determine  capacities  of  system 

INCOSE  Pg4.3 

100% 

TP-1 

3 

Determine  concept  of  operations 

INCOSE  Pg  4.3 

OoDAf  OV  1.  CONORS. 

Scenarios 

60% 

TP  1 

4 

Utilue  scenarios  to  define  operations  concept  documents 

1  NCOS'  Pg  4.4 
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TT»1 

4 

Determine  response  to  undesred  inputs 

Buede  Pg  131 

TP  1 

4 

Determine  response  to  unexpected  inputs 

Buede  Pg  131 

TP-1 

4 

Determine  bounds  on  expected  inputs 

Buede  Pg  131 

TP  1 

4 

Determine  aporopnjte  response  on  expected  inputs 

Buede  Pg  131 

TP  1 

3 

Identify  output  requirements 

Buede  Pg  4b 

lOEFO,  Use  Cases 

100% 

TP  1 

4 

Identify  how  test  data  obtained  for  output  reqs 

Buede  Pg  4$ 

TP-1 

4 

Determine  accuracy  of  output 

Buede  Pg  131 

TP  1 

4 

Determine  correctness  of  output 

Buede  Pg  131 

TP-1 

4 

Determine  secunty/surwabriity  of  output 

Buede  Pg  131 

TP  1 

4 

Determine  intensity/stze  of  output 

Buede  Pg  131 

TP  1 

4 

Determine  number  per  unit  time  of  output 

Buede  Pg  131 

TP  1 

4 

Determine  coverage  of  output 

Buede  Pg  131 

TP  1 

4 

Determine  response  time  of  output 

Buede  Pg  131 

TP  1 

4 

Determine  update  frequency  of  output 

Buede  Pg  131 

TP  1 

4 

Determine  availability  of  output 

Buede  Pg  131 

TP-1 

3 

Determine  verification  criteria 

Bn*P|23  MCOa 

Pg  3-4 

System  Requirements 
Document.  Test  & 

{valuation  Master  Plan 

30% 

TP  1 

4 

Identify  how  test  data  determines  system  conforms  to  design 

Buede  Pg  4S 

TP-1 

4 

Identify  how  test  data  determ  nes  system  acceptable  to  stakeholders 

Buede  Pg  4S 

TP-1 

4 

Identify  how  test  data  obtained  for  technology  and  system  wide  reqs 

Buede  Pg  4b 

TP  1 

3 

Determine  validation  criteria 

Buede  Pg  23.  INCOSC 
Pg4J 

System  Requirements 
Validation  Document. 

Test  &  Evaluation 

Master  Plan 

30% 

TP-1 

4 

Identify  how  test  data  determine  system  complies  with  originating  reqs 

Buede  Pg  4b 

TP  1 

4 

Identify  how  test  data  determ  nes  system  acceptable  to  stakeholders 

Buede  Pg  4b 

TP  1 

4 

Identify  how  test  data  obtained  for  system  wide  reqs 

Buede  Pg  4b 

TP  2 

1 

TP  2  (Logical  Analysis) 

S%  100% 

TP  2 

2 

Analysis  Preparation 

DAG  Sec  4 .2.4 2. 

100%  100% 

TP  2 

3 

Identify  logical  groupings  of  elements 

DAG  bee  4  2  4-2. 

100% 

TP  2 

2 

Perform  Functional  Analysis 

DAG  Sec  4.24.2. 

OoDAFOV  S.SV  b 

100%  100% 

TP  2 

3 

Identify  system  boundaries 

INCOSE  Pg  4.b 

Contort  Diagram 

100% 

TP  2 

4 

Identify  functional  boundaries 

INCOG*  Pg  4.fa 

TP-2 

3 

Identify  functional  reiationshps  |lCOMs| 

DAG  Sec  4.2.4 2. 

COAE.  System  Architect 

100% 

TP  2 

4 

Identify  Inputs  of  Functions 

INCOS*  Pg  4.S 

TP  2 

S 

Utilize  stakeholder  requirements 

INCOG*  Pg4.S 

TP  2 

s 

Utilize  cost  constraints 

INCOS*  Pg  4.S 

TP  2 

s 

Utilize  schedule  constraints 

INCOS*  Pg  4.S 

TP  2 

s 

Utilize  solution  constraints 

INC  OSS  Pg  4.S 

TP  2 

s 

Utilize  traceabikty  matro 

INCOG*  Pg  4.S 

TP  2 

4 

Identify  Controls  of  Functions 

INCOS*  Pg4.S 

TP  2 

s 

Utilize  terms  and  conditions  of  agreements 

INCOS*  Pg  4.S 

TP  2 

s 

Utilize  project  procedures  and  processes 

INCOS£  Pg4.S 

TP  2 

4 

Identify  Outputs  of  Functions 

INCOS*  Pg  4.S 

TP  2 

s 

Derrve  addtional  requrements 

INCOSi  Pg4.S 

TP  2 

s 

Define  functional  boundaries 

INCOS*  Pg  4.S 

TP  2 

Identify  interfaces 

INCOS*  Pg4.S 

TP  2 

& 

Identify  ^formation  exchange  requirements  with  systems  external 
to  functional  boundaries 

INCOS*  Pg  4.4,  4.S 

TP  2 

s 

Determine  functional  requirements 

DAG  Sec  4 .2.4.2.; 
INCOS*  Pg  4.S 

TP-2 

s 

Determine  performance  requrements 

DAG  Sec  4 .2.4-2.; 

INC  OS*  Pg  4.S 

TP  2 

s 

Determine  non  functional  requrements 

DAG  Sec  4 .2.4.2.; 
INCOSt  Pg4.S 

TP  2 

Determine  architectural  constraints 

INCOS*  Pg  4.S 

TP  2 

4 

Identify  Mechanisms  of  Fimctions 

INC  OS*  Pg  4.S 
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TP  2 

Identify  physical  des  <n  constraints 

INCOG*  Pg  4.6 

TP-2 

S 

Utilize  enterprise  infrastructure 

INCOS*  Pg  4.S 

TP-2 

s 

Utilize  enterprise  pokies.  procedures,  and  standards 

INCOSt  Pg  4.5 

OoDAf  TV  1  (Standards 
ProfiieJ.  TV  2 
•  Standards  Forecast} 

TP  2 

3 

Allocate  functional  requirements 

DAG  Sec  A.2.A2. 

100% 

TP  2 

4 

Define  functional  performance 

INCOS*  Pg  A.b 

TP  2 

Determine  measures  of  performance 

INC  OS*  Pg  4.6 

TP  2 

Determine  measures  of  effectiveness 

INCOSt  Pg  A.b 

TP  2 

3 

Allocate  performance  parameters 

DAG  Sec  4.2  4-2. 

100% 

TP  2 

3 

Allocate  performance  constraints 

DAG  Sec  4  2  4-2. 

100% 

TP  2 

2 

Perform  Behavioral  Analyses 

DAG  Sec  4.2  4.2. 

OoDAf  OV  Gja.b.cl  SV 
lOla^c) 

100%  100% 

TP-2 

3 

Identify  behavioral  relationships 

OAG  Sec  A2.A2. 

Data  Flow  Diagrams, 
UMl 

100% 

TP  2 

A 

Perform  data  flow  analysis 

DAG  Sec  4.2  4.2. 

TP  2 

A 

Perform  object  oriented  analysis 

DAG  Sec  4  2  4-2. 

TP  2 

3 

Identify  tempora  relationships 

OAG  Sec  4.2.4 2. 

OV  6c 

100% 

TP  2 

A 

Perform  bmeHne  analysis 

DAG  Sec  4.2.4-2. 

TP-2 

3 

Identify  key  interfaces 

OAG  Sec  4.2  4.2. 

trtemai  Systems 
Diagram 

100% 

TP  2 

A 

Identify  functional  interfaces 

Student  Derived 

TP  2 

Identify  functcnal  interfaces  to  interacting  systems 

INC  OS*  Pg  4.b 

TP  2 

Identify  fun  clonal  interfaces  to  interacting  platforms 

INCOS*  fg4.b 

TP  2 

Identify  functional  interfaces  to  interacting  humans 

INC  OS*  Pg  4.b 

TP  2 

A 

Identify  design  interfaces 

Student  Derived 

TP  2 

s 

Identify  des-gn  interfaces  to  interacting  systems 

INC  OS*  Pg  4.b 

TP  2 

s 

Identify  design  interfaces  to  interactng  platforms 

INC  OS*  Pg  4.b 

TP  2 

s 

Identify  design  interfaces  to  interactng  humans 

INC  OS*  Pg  4.b 

TP  2 

4 

Define  interfaces 

OAG  Sec  4 .2.4.2. 

interlace  Control 

Document 

TP  2 

2 

Perform  Environmental  Analysis 

DAG  Sec  4.2.4.2. 

10%  100% 

TP  2 

3 

Identify  natural  environmental  factors 

OAG  Sec  4.2  4.2. 

100% 

TP-2 

4 

Identify  natural  environmental  factors  affecting  system  performance 

INCOG*  Pg4.b 

TP  2 

4 

Identify  natixal  environmental  factors  impacts  Inman  comfort 

INCOG  t  *g4.b 

TP  2 

4 

Identify  natiaal  environmental  factors  impact^  hirnan  safety 

INC  OS*  Pg  4.b 

TP  2 

4 

Identify  natiaal  environmental  factors  causing  human  error 

INC  OS*  Pg  4.b 

TP  2 

3 

Identify  Induced  environmental  factors 

DAG  Sec  4.2  4.2. 

100% 

TP-2 

4 

Identify  induced  environmental  factors  affeebng  system  performance 

INCOG*  Pg  A.b 

TP  2 

4 

Identify  induced  environmental  factors  impacting  human  comfort 

INCOG*  pg  4.b 

TP  2 

4 

Identify  induced  environmental  factors  impacting  human  safety 

INCOG*  Pg4.b 

TP  2 

4 

Identify  induced  environmental  factors  causing  human  error 

INCOSt  3g4.b 

TP  2 

4 

Identify  potential  environmental  impact 

INCOSt  3g  4.b 

TP  2 

3 

Identify  training  requirements 

INCOS*  Pg  4.10 

10% 

TP  2 

3 

Identify  human  systems  integration  requirements 

INCOG*  Pg  4.b 

100% 

TP  2 

3 

Identify  system  security  reqiarements 

INCOG*  Pg  4.b 

100% 

TP  2 

2 

Design  Factors  Analysis 

INCOG*  Pg  4.b 

S%  100% 

TP  2 

3 

Identify  production  design  factors  to  facilitate  efficient  functions 

INCOS*  Pg  4.b 

60% 

TP-2 

3 

Identify  deployment  design  factors  to  facilitate  efficient  functions 

INCOG*  Pg  4.b 

30% 

TP  2 

3 

Identify  transten  design  factors  to  facilitate  efficient  functions 

INCOS*  »g4.b 

100% 

TP  2 

3 

Identify  operation  design  factors  to  facilitate  efficient  functions 

INCOSt  pg 4.b 

60% 

TP  2 

3 

Identify  maintenance  design  factors  to  facilitate  efficient  functions 

INCOS*  Pg  4.b 

30% 

TP  2 

3 

Identify  re  engineering  design  factors  to  faaiitate  effie  ent  functions 

INCOG*  Pg  A.b 

so% 

TP  2 

3 

Identify  dsposal  design  factors  to  facilitate  efficient  functions 

INCOS-*  Pg  4.b 

s% 

TP  2 

2 

Develop  Functional  Architecture 

OAG  Sec  4.24.2. 

50%  100% 

TP  2 

3 

Document  functional  architecture 

DAG  Sec  4.2  4.2. 

OoDAf  OV  S.  SV  4,S 

100% 

TP  2 

3 

Identify  technology  alternatives 

DAG  Sec  4.2.4^. 

100% 

TP-2 

3 

Identify  recogivred  standards  to  be  used  in  functional  architecture 

OAG  Sec  4.2.4.2.; 
INCOG*  Pg  4.6 

100% 

TP  2 

3 

Ma»ntam  configuration  control  of  functions  architecture 

INCOSt  Pg  4.b 

50% 

TP  2 

3 

Define  criteria  to  verify  functional  architecture 

INCOG*  Pg  4.6 

100% 
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TP-3 

1 

TP- J  {Dew  Solution) 

30%  100% 

TP  3 

2 

Define  Design  Problem 

Bjede  Pg  31.  39 

SO%  100% 

TP  3 

3 

Utilize  stakeholder  nputs 

Buede  Pg  39,  40. 12S 

100% 

TP  3 

3 

UbAze  orignating  requirements 

Buede  Pg  39 

100% 

TP  3 

4 

Utilize  system  requirements 

Buede  Pg  39 

TP-3 

3 

1/tJkze  operational  concept 

Bjede  Pg  39.  40 

so% 

TP  3 

3 

Utdze  system  nputs,  controls,  and  outputs 

Buede  Pg  40 

100% 

TP  3 

4 

Utilize  system  i routs 

Buede  Pg40 

TP  3 

S 

Utilize  functional  requirements 

I  NCOS*  Pg  4.7 

TP  3 

Utilize  performance  requre merits 

INC  OS*  Pg  4.7 

TP  3 

Utilize  architectural  constraints 

INC  OS*  Pg4.7 

TP  3 

Utilize  traceabikty  matrw 

INC  OS*  Pg  4.7 

TP  3 

Utilize  system  solution  constraints 

1  NCOS'  Pg  4.7 

TP-3 

4 

Utilize  system  controls 

INC  OS*  Pg4.7 

TP  3 

Utilize  natural  and  societal  laws 

INCOS*  Pg  4.7 

TP  3 

4 

Utilize  system  outputs 

Buede  Pg  4 0 

TP  3 

Determine  system  element  detailed  descriptions 

Bjede  Pg  AO 

TP  3 

Assign  requirements  to  system  elements 

Bjcde  Pg  AO 

TP  3 

Document  requirement  assignment  to  system  elements 

Buede  Pg  40 

TP  3 

Determine  interface  requirements 

Bjede  Pg  40 

TP  3 

Establish  plan  for  system  ntegrabon 

Buede  Pg  40 

TP  3 

Establish  plan  for  verification  strategy 

Buede  Pg  40 

TP-3 

4 

Utilize  system  mechanisms 

Buede  Pg  40 

TP  3 

3 

Ubkze  trade  studes  analysis  tools 

Buede  Pg  40 

Objectives  Hierarchy, 
Quality  Function 
Deployment,1’ Mo  use  of 
Quality,  Influence 
Diagrams 

so% 

TP  3 

4 

Utilize  SE  team  input 

Buede  Pg  40 

TP  3 

3 

Define  system  botndanes 

Buede  Pg  40 

OoDAf  SV  1 

100% 

TP-3 

4 

Identify  interfaces  Oe tween  system  cements  and  external  systems 

TP  3 

4 

Utilize  system  boundaries 

Buede  Pg  40 

TP-3 

3 

Parte  ion  system  requirements 

OoDAf  SV  4.S 

80% 

TP  3 

4 

Allocate  system  requirements  to  system  dements 

TP  3 

3 

Define  system  integration  strategy 

80% 

TP  3 

2 

Generate  Alternative  Design  Solutions 

Buede  Pg  31.  39 

SS%  100% 

TP  3 

3 

Identify  aporopnate  products 

DAG  Sec  4.2.4  J. 

100% 

TP  3 

4 

Identify  products 

DAG  Sec  4.2.43. 

TP  3 

4 

Identify  bass  for  work  breakdown  structures 

DAG  Sec  4.2.43. 

TP  3 

4 

Identify  basts  for  basdines 

DAG  Sec  4.2.43. 

TP-3 

3 

Develop  performance  attributes  and  measures 

DAG  Sec  43.43. 

Technical  Performance 

Measures 

100% 

TP  3 

4 

Oevdop  specifications 

DAG  Sec  4.2.43. 

TP  3 

4 

Identify  basts  for  specifications 

DAG  Sec  4.2.43. 

TP-3 

3 

Develop  physical  baseline  architecture 

DAG  Sec  4.2.43.; 

Buede  Pg  31.  39 

100% 

TP  3 

4 

Develop  functional  architecture 

Buede  Pg  31.  39 

TP  3 

4 

Oevdop  operational  architecture 

Buede  Pg  31.  39 

TP  3 

4 

Address  development  system 

Buede  Pg  39 

TP  3 

4 

Address  manufacturing  system 

Buede  s»g  39 

TP  3 

4 

Address  deployment  system 

Buede  Pg  39 

TP  3 

4 

Address  training  system 

Buede  Pg  39 

TP  3 

4 

Address  refinement  system 

Buede  Pg  39 

TP  3 

4 

Address  retirement  system 

Buede  Pg  39 

TP  3 

3 

UbAze  prototypes 

(NCOS*  Pg  3.7 

70% 

TP  3 

4 

Utilize  prototypes  to  verify  concept  feasibly 

INCOS*  Pg  3.7 

TP  3 

4 

Utilize  prototypes  to  eipiorc  risks 

INCOS*  Pg  3.7 

TP  3 

4 

Utilize  prototypes  to  e«p*ore  opportunities 

INCOS*  Pg  3.7 

TP  3 

4 

Utilize  prototypes  to  eiplore  affordability  assessment 

INCOS*  Pg  3.7 

TP  3 

4 

Utilize  prototypes  to  eiplore  environmental  impact 

INCOS*  Pg  3.7 

TP  3 

4 

Utilize  prototypes  to  eiplore  opportunities 

INCOS*  Pg  3.7 

TP  3 

4 

Utilize  prototypes  to  eiplore  fadure  modes 

INCOS'  Pg  3.7 

TP  3 

4 

Utilize  prototypes  to  eiplore  hazard  analysis 

INCOS*  Pg  3.7 

TP  3 

4 

Obtain  confidence  solution  a  achievable 

INCOS*  Pg  3.7 

TP  3 

4 

Identify  potential  component  problems 

INCOS*  Pg  3.7 

TP  3 

3 

Provide  design  outputs 

Buede  Pg  40 

100% 

TP  3 

4 

Provide  output:  system  boundaries 

Buede  Pg  40 
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TP  3 

4 

Provide  output:  inputs 

Bucde  ?g  40 

TP  3 

4 

Provide  output:  outputs 

Buede  Pg  AO 

TP  3 

4 

Provide  output:  qualrficaton  par 

Buede  Pg  39 

TP  3 

4 

Provide  output:  design  documentation 

Buede  Pg  39 

TP  3 

4 

Provide  output:  operational  concept 

Buede  Pg  40 

TP  3 

4 

Provide  output:  objectives  hierarchy 

Buede  Pg  40 

TP  3 

4 

Provide  output:  originating  requirements 

Buede  Pg  40 

TP  3 

4 

Provide  output:  system  requirements 

Buede  Pg  40 

TP  3 

4 

Provide  output:  design  feasibility 

Buede  Pg  40 

TP  3 

4 

Provide  output:  test  requirements 

Buede  Pg  40 

TP-3 

3 

Define  ogical  architecture 

INCOSE  Pg  4.8 

Data  Structure.  Rules 
Modd  <OV  ba.  SV  10a  | 

SS% 

TP  3 

4 

Define  data  structure 

Student  Derived 

TP  3 

4 

Define  ndes  model 

Student  Derived 

TP  3 

2 

Evaluate  Design  Alternatives 

DAG  Sec  4.2.43 

30%  100% 

TP  3 

3 

Perform  concept/dcs^n  analysts 

DAG  Sec  4.2.43. 

100% 

TP  3 

3 

Consider  existing  off  the  shelf  solutions 

INCOSE  Pg  4.8 

100% 

TP  3 

3 

Evaluate  alternative  design  solutions 

INCOSE  »g  4.8 

80% 

TP  3 

3 

Identify  recogrvzed  standards  to  be  used  in  design  solution 

INCOSE  Pg  4.b 

OoDAF  TV  1 

60% 

TP  3 

3 

Perform  trade  studies 

DAG  Sec  4.2.43. 

80% 

TP  3 

4 

Perform  operational  capabilities  trade  studies 

DAG  Sec  4.S.6 

TP-3 

4 

Perform  functional  trade  studies 

DAG  Sec  4.S.6 

TP  3 

4 

Perform  manufactunng  trade  studes 

DAG  Sec  4.S.6 

TP  3 

4 

Perform  testing  trade  studies 

DAG  Sec  4.S.6 

TP  3 

4 

Perform  support  trade  studies 

DAG  Sec  4.S.6 

TP  3 

4 

Perform  lifecyde  cost  trade  studies 

OAG  Sec  4.S.6 

TP  3 

3 

Perform  design  mode  ns 

DAG  Sec  4.2.43. 

so% 

TP  3 

4 

Develop  environmental  modeling  view 

Buede  30 

TP  3 

4 

Develop  data/informationaJ  modeling  view 

Buede  Pg  30.  INCOSE 

Core  Architecture  Data 
Model,  SysML  |  Logical 
Architecture) 

TP-3 

4 

Develop  process  modeling  view 

BucOcPg  JO 

TP  3 

4 

Develop  behavior  modeling  view 

Buede  =*g  30 

TP  3 

S 

Consider  feature  interactions 

INCOSE  pg  4.8 

TP-3 

s 

Consider  emergent  properties 

INCOSE  Pg  4.8 

TP  3 

s 

Consider  human  machine  interactions 

INCOSE  Pg  4.8 

TP  3 

4 

Develop  implementation  modeling  view 

Buede  Pg  30 

TP  3 

4 

Validate  simulations 

INCOSE  Pg  4.11 

TP  3 

3 

Evaluate  to  performance  attributes  and  measures 

DAG  Sec  4.2.43. 

100% 

TP  3 

4 

Complete  product  analyses 

INCOSE  Pg  4.10 

TP  3 

4 

Complete  process  analyses 

INCOSE  Pg4.10 

TP  3 

4 

Complete  material  analyses 

INCOSE  Pg  4.10 

TP  3 

3 

Confirm  nteroperaoility 

OAG  Sec  4.2.43. 

30% 

TP4 

1 

S%  100% 

TP  4 

2 

INCOSE  Pg4.10 

30%  100% 

TP  4 

3 

Utilize  des<gn  requirements 

INCOSE  **g 4.9 

100% 

TP  4 

3 

Utilize  verification  criteria 

INCOSE  Pg  4.9 

100% 

TP  4 

3 

Utilize  validation  catena 

INCOSE  Pg  4.9 

100% 

TP  4 

3 

Utofeze  terms  and  condtions  of  agreements 

INCOSE  Pg  4.9 

100% 

TP  4 

3 

Ubize  government  and  odustry  standards 

INCOSE  Pg  4.9 

M  L  STD.  IEEE.  ISO 

100% 

TP  4 

3 

Improve  process  control  with  Lean  Design 

INCOSE  Pg  4.10 

30% 

TP  4 

4 

Inspections  are  proactive  way  to  bold  in  quality 

INCOSE  Pg  4.10 

TP  4 

2 

Fabricate  Hardware 

DAG  Sec  4.2.4  A. 

30%  100% 

TP  4 

3 

Develop  deta.ied  drawings 

INCOSE  Pg  4.10 

100% 

TP  4 

3 

Identify  hardware  conjuration  items  for  assembly 

l Student  Delved} 

80% 

TP4 

3 

Identify  implementation  tolerances 

INCOSE  Pg4.10 

SO% 

TP  4 

3 

Develop  deta.ied  matcnal  specifications 

INCOSE  Pg  4.10 

30% 

TP  4 

3 

Define  fabrication  procedures 

INCOSE  Pg  4.10 

100% 

TP  4 

4 

Identify  tools 

INCOSE  Pg  4.10 

TP  4 

4 

Identify  equipment 

INCOSE  Pg  4.10 

TP  4 

4 

Assure  consistent  and  repeataole  clement  production 

INCOSE  Pg  4.10 

TP4 

2 

Code  Software 

DAG  Sec  4.2.4  4. 

70%  100% 

TP  4 

3 

Identify  software  configuration  terns  for  compiling 

(Student  Delved ) 

100% 

TP  4 

3 

Define  coding  procedures 

INCOSE  Pg  4.10 

70% 
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TP  4 

3 

□eve-op  detailed  codes 

INCOSE  Pg  4.10 

100% 

TP  4 

2 

Conduct  Unit  Testing 

IhCQSE  Pg  4.10 

50%  100% 

TP  4 

3 

Generate  acceptance  test  proeedires 

DAG  Sec  4.2.4  4. 

60% 

TP  4 

3 

Inspect  hardware  and  software  for  correct  functionality 

INCOSE  Pg4.10 

System  functionality 
Descncbon  (S V  4) 

100% 

TP  4 

3 

Test  hardware  and  software  for  correct  functionality 

INCOSE  Pg  4.10 

100% 

TP  4 

4 

Perform  white  bon  testing  on  software 

INCOSE  Pg4.10 

TP  4 

3 

Ensure  sufficient  hardware  and  software  testing  poor  to  integration 

INCOSE  Pg  4.11 

SO% 

TP  4 

2 

Conduct  Training 

INCOSE  Pg  4.10 

5%  •  S% 

TP  4 

3 

Establish  initial  staff  of  trained  users 

INCOSE  Pg4.10 

TP  4 

3 

Establish  in  nal  staff  of  trained  mam  tamers 

INCOSE  Pg4.10 

TP  4 

3 

T  ram  initial  operators 

INCOSE  Pg  4.10 

S% 

TP  4 

3 

T  rain  initial  mamtamers 

INCOSE  Pg  4.10 

S% 

TP  4 

2 

Prepare  for  Integration 

□AG  Sec  4.2.4  4. 

80%  100% 

TP  4 

3 

Supply  system  element  for  venftcation  and  validation 

INCOSE  Pg  4.10 

80% 

TP4 

3 

Refine  integration  constraints 

INCOSE  Pg  4.10 

100% 

TPS 

1 

TP-5  (Integration) 

10%  100% 

TPS 

2 

Determine  Integration  Process 

Buede  Pg  310 

100%  100% 

TPS 

3 

Execute  assembly  processes  /  procedures 

DAG  Sec  4.2.4.S. 

100% 

TPS 

4 

Define  integration  strategy  regard ng  ava  lat*  ty  of  system  elements 

INCOSE  Pg  4.12 

TPS 

4 

Determine  assembly  sequence 

DAG  Sec  4.2.4-S. 

TPS 

4 

Identify  assembly  const ramts 

DAG  Sec  4.2.4.S. 

TPS 

4 

Utilize  top  down  process 

Buede  Pg  310 

TPS 

Select  top  level  module,  with  simulated  components 

Buede  Pg  310 

TPS 

S 

Replace  samdated  components  with  actual,  one  at  a  bme,  to  qualify 
top  level  module 

Buede  Pg  310 

TPS 

4 

Utilize  bottom  up  process 

Bjede  Pg  310 

TPS 

S 

Test  components*  mdikvdualiy 

Buede  Pg  310 

TPS 

s 

Test  assembled  components  until  enbre  system  assembled  and 
tested 

Buede  Pg  310 

TPS 

4 

Utilize  big  bang  process 

Buede  Pg  310 

TPS 

S 

Assemble  untested  components 

Buede  Pg  310 

TPS 

s 

Test  assembly 

Buede  Pg  310 

TPS 

3 

□eve-op  integration  plan 

INCOSE  Pg  4.12 

100% 

TPS 

2 

Conduct  Assembly  /  Integration  of  System 

INCOSE  Pg4.12 

10%  100% 

TPS 

3 

Ubize  configurator!  item  components 

Buede  Pg  42 

Configuration  Items 

40% 

TPS 

4 

Utilize  hardware  configuration  item  components 

('Student  Derived) 

TPS 

4 

Utilize  software  configuration  item  components 

(Student  Derived) 

TPS 

3 

Ubize  integration  technology  constraints 

INCOSE  Pg4.12 

10% 

TPS 

3 

Ubize  architectural  design  reqiarements 

INCOSE  Pg4.12 

100% 

TPS 

3 

Assemble  system  elements  according  to  integration  plan 

INCOSE  Pg  4.12 

100% 

TPS 

4 

Utilize  integration  tools 

INCOSE  Pg4.12 

TPS 

s 

Schedide  rvtegrabon  testing  tools 

INCOSE  Pg4.12 

TPS 

4 

Utilize  integration  faciktos 

INCOSE  pg4.12 

TPS 

S 

Schediae  rdegration  testing  facilities 

INCOSE  Pg  4.12 

TPS 

4 

Utilize  integration  test  eqiapment 

INCOSE  Pg4.12 

TPS 

3 

Obtain  subsystem  /  system  ready  for  verificaton 

INCOSE  Pg4.12 

80% 

TPS 

3 

Conduct  integration  verification  testirg 

Buede  Pg  42 

80% 

TPS 

4 

Utilize  acceptance  test  plan 

Buede  Pg  42 

TPS 

4 

Inspect  to  verrficjbon  reqiarements 

Buede  Pg  32 

TPS 

3 

Validate  interna  -rterfaces 

INCOSE  Pg  4.12 

OoDAf  SV  1 

100% 

TPS 

4 

Validate  internal  interfaces  through  Wadi  bon  testing  at  each  assembly 
level 

INCOSE  Pg  4.12 

TPS 

4 

Confirm  correct  functianafety  of  assembled  products  at  each  assembly 
level 

INCOSE  Pg  4.12 

TP  S 

3 

Conduct  early  evaluation  agansr  performance  attributes 

Buede  Pg  42 

100% 

TPS 

4 

Utilize  stakeholder  inputs 

Buede  Pg  42 

TPS 

4 

Utilize  operational  concept 

Buede  Pg  42 

TPS 

4 

Utilize  originating  reqiarements 

Buede  Pg  42 

TPS 

4 

Ublizc  (Served  requirements 

Buede  Pg  42 

TPS 

3 

Obtain  integration  test  and  ana  yvs  results 

INCOSE  Pg4.12 

100% 

TPS 

3 

Determine  adjustments  to  system  elements 

DAG  Sec  4.2.4  4. 

100% 
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TPS 

2 

Relevant  Environment 

0AGSec4.2.4-S. 

40%  40% 

TPS 

3 

Incorporate  assembled  system  into  relevant  environment 

DAG  Sec  4.2.44. 

40% 

TP* 

1 

TP  6  (Venficatiofi) 

80%  100% 

TP* 

2 

Plan  Verification 

Buede  Pg  314 

80%  1 00% 

TP* 

3 

Review  /  utilize  system  objectives 

Buede  Pg  314.  iNCOSt 
P*4.13 

System  Requirements 
Document.  Test  & 

Evaluaton  Master  Plan 

100% 

TP* 

3 

Review  /  utilize  allocated  requirements  to  functional  /  physical  /  logical 
architectures 

Buede  Pg  314 

100% 

TP* 

4 

Utilize  functional  architectures  for  verification  components 

Buede  Pg  314 

TP* 

A 

Utilize  derived  requrements  to  functional  architectures 

Suede  Pg  314 

TP* 

A 

Utilize  functions  to  physical  architectures 

Buede  Pg  314 

TP* 

3 

Create  master  verification  plan 

Buede  Pg  314 

80% 

TP* 

4 

Create  test  scenarios 

Buede  Pg  314 

TP* 

A 

Write  activity  level  verification  plans  for  each  verification  component 

Buede  Pg  314 

TP* 

S 

Identify  required  simulation  data  for  each  actwitv 

Buede  Pg  314 

TP* 

3 

Develop  detailed  derived  verification  procedures 

Buede  Pg  314,  MCOSE 
Pg  4.14 

80% 

TP* 

4 

Utilize  verification  criteria  (to  include  pass/fail  thresholds)  and  methods 
(analysis,  inspection,  demonstration.  &  testing | 

Buede  Pg  314,  INCQSE 
PR  4.13 

TP* 

4 

Utilize  verification  criteria  and  method  for  system  objectives 

Buede  Pg  314 

TP* 

4 

Utilize  verification  criteria  and  method  for  operational  concept 

Buede  Pg  314 

TP* 

4 

Utilize  acceptance  test  plan 

Buede  Pg  42 

TP* 

3 

Write  test  procedures 

Buede  Pg  314 

100% 

TP* 

3 

Write  analysis  procedures 

Buede  Pg  314 

80% 

TP* 

3 

Assign  verrficatnn  responsbilrbes 

Buede  Pg  314 

100% 

IP* 

4 

Assam  vertf  tcation  actmtw*  to  organization* 

buede  314 

TP* 

4 

Allocate  verification  activities  to  resources 

Buede  Pg  314 

TP* 

2 

Execute  Verification 

INCQSr  •'g  4.14 

80%  100% 

TP* 

3 

Verify  materials  are  safe 

DAG  Sec  4.2.4  *. 

100% 

TP* 

3 

Demonstrate  system 

DAG  Sec  4.2.4 *.; 

Buede  Pg  SO 

100% 

TP* 

4 

Perform  modekrg  &  simulation 

DAG  Sec  4.2.4*.; 

Buede  Pg  SO 

TP* 

4 

Ensure  interfaces  operational  when  older  component  replaced  with 
newer  component 

INCQSE  Pg  2.S 

TP* 

3 

Conduct  verification  testng 

Buede  Pg  42 

100% 

TP* 

4 

Test  to  verification  requirements 

Buede  Pg  32 

TP* 

4 

Test  at  lowest  system  element  level 

DAG  Sec  4.2.4*.; 

Buede  Pg  32.  INCOSt 
Pg  4  13 

TP* 

4 

Test  to  sub  system  level 

DAG  Sec  4.2.4*.; 

Buede  Pg  42 

TP* 

S 

Record  evidence  that  system  element  satisfies  requrements,  or  not 

INCQSE  Pg  4.14 

TP* 

4 

Test  to  system  level 

OAG  Sec  4.2.4*.; 

Buede  Pg  42 

TP* 

S 

Record  evidence  that  system  satisfies  requrements,  or  not 

INCQSE  ^g  4.14 

TP* 

4 

Perform  software  component  testing 

DAG  Sec  4.2.4*. 

TP* 

3 

Document  verification  testing 

Buede  Pg  42 

100% 

TP* 

4 

Provide  output:  accepted  system 

Buede  Pg  42 

TP* 

3 

Inspect  to  verification  requirements  and  tolerances 

OAG  Sec  4.2.4*.; 

Buede  Pg  32,  SO 

80% 

TP* 

3 

Identify  deficiencies 

Buede  Pg  32 

100% 

TP* 

4 

Record  recommended  corrective  actions 

INCQSE  Pg  4.14 

TP* 

3 

Correct  deficiencies 

100% 

TP* 

4 

Record  corrective  actions  taken 

INCQSE  Pg  4.14 

TP* 

3 

Document  uncorrectabk?  defioenoes 

Buede  Pg  32 

100% 

TP* 

3 

Analyze  test  results 

DAG  Sec  4.2.4*. 

80% 

TP* 

3 

Document  verification  resists 

Buede  Pg  42.  INCOSt 
PR  4.14 

80% 

TP* 

3 

Provide  output  verified  components  and  system 

Buede  Pg  42 

100% 

TP* 

3 

T ransfer  verified  system  to  validation  testeg 

Buede  Pg  42 

80% 

100 


TP  7 

1 

TP  7  (Validation) 

80%  100% 

TP-7 

2 

Plan  Validation 

Boede  Pg  314 

80%  10O% 

TP-7 

3 

Review  /  utilize  system  objectives 

Buede  Pg  314 

System  Requirements 
Document,  Test  & 

Evaluaton  Master  Plan 

100% 

TP  7 

4 

Identify  validation  system  objectives 

&uede  Pg  314 

TP  7 

S 

Utilize  valdatwr  operational  concept 

Buede  Pg  314 

TP  7 

S 

Utilize  vakdatnn  requirements 

Suede  Pg  314 

TP  7 

Identify  pass/fad  thresholds 

Buede  Pg  314 

TP  7 

3 

Define  validation  architectures 

Buede  Pg  314 

80% 

TP  7 

4 

Define  validation  functional  architecture 

Buede  Pg  314 

TP  7 

S 

Allocate  requirements  to  functional  architecture 

Buede  Pg  314 

TP  7 

4 

Define  validation  generic  physical  architecture 

Bjede  Pg  314 

TP  7 

S 

Allocate  functions  to  generic  phyvcal  architecture 

Buede  Pg  314 

TP  7 

4 

Oeveiop  functional  architectures  for  validation  components 

Buede  Pg  314 

TP  7 

S 

Allocate  denved  reqiarements  to  functional  architectures 

Buede  Pg  314 

TP  7 

s 

Allocate  f  unctions  to  physical  architectures 

Buede  Pg  314 

TP  7 

3 

Create  master  validation  plan 

Buede  Pg  314 

80% 

TP-7 

4 

Write  activity  eve  validation  plans  for  each  validation  component 

Buede  Pg  314 

TP  7 

S 

Utilize  vakdation  criteria  far  stakeholder  requirements 

INCOSE  Pg4.l7 

TP  7 

4 

Identify  required  snulattcn  data  for  each  activity 

Buede  Pg  314 

TP-7 

4 

Create  test  scenarios 

Buede  »g  314 

TP  7 

S 

Utilize  scenarios  exercising  at  system  modes 

INCOSE  Pg4.l7 

TP-7 

3 

Write  test  procedures 

Buede  Pg  314 

100% 

TP-7 

3 

Writ  analysts  procedures 

buede  Pg  J14 

HAT* 

TP  7 

4 

Develop  validation  procedures  to  show  system  fit  for  its  purpose 

INCOSE  Pg4.l7 

TP-7 

4 

Develop  validation  procedures  to  show  system  satisfies  stakeholder's 
requirements 

INCOSE  Pg4.l7 

TP-7 

3 

Assign  validation  responsibilities 

Buede  Pg  314 

100% 

TP  7 

4 

Assgr  validation  activities  to  organizations 

Buede  Pg  314 

TP-7 

S 

Test  with  ultimate  users 

DAG  Sec  4.2.47.; 

Buede  Pg  SI 

TP  7 

4 

Allocate  validation  activities  to  resources 

Buede  Pg  314 

TP  7 

S 

Test  in  operational  environment 

DAG  Sec  4.2.4  7. 

TP-7 

2 

Execute  Validation 

Buede  Pg  SI.  INCOSE 
Pg4.l7 

80%  100% 

TP-7 

3 

Utilize  integrated  system  released  for  validation 

Buede  Pg  42.  INCOSE 
Pg  4.17 

80% 

TP  7 

4 

Obtan  approved  system  baseline 

INCOSE  Pg4.l7 

TP  7 

3 

Ensure  system  readness  to  conduct  vakdation 

INCOSE  Pg  4.17 

100% 

TP  7 

4 

Ensure  enabling  system  readiness  to  conduct  validation 

INCOSE  Pg4.17 

TP  7 

4 

Ensure  trained  operator  readiness  to  conduct  validation 

INCOt!  *g4.l7 

TP  7 

3 

Perform  umuation 

DAG  Sec  4.2.4  7. 

100% 

TP  7 

4 

Prove  in  with  prototypes 

DAG  Sec  4.2.4  7. 

TP  7 

4 

Prove  in  with  mode  ups 

DAG  Sec  4.2.47. 

TP  7 

4 

Prove  in  with  modeling  &  simulation 

DAG  Sec  4.2.47. 

TP  7 

3 

Conduct  system  validation  te strip 

Buede  Pg  42 

10 0% 

TP-7 

4 

Demonstrate  system  level  performance  over  entire  operating  regime 

INCOSE  Pg4.l7 

TP-7 

4 

Utilize  results  to  correct  performance  deficiencies  before 
implementation 

INCOSE  Pg4.l7 

TP  7 

3 

Analyze  vakdaten  test  resists 

INCOSE  Pg4.17 

100% 

TP  7 

4 

Obtan  resists  of  validation  activities 

INCOSE  Pg  4.17 

TP  7 

4 

Detect  trends  in  failure 

INC  OS?  pg4.l7 

TP  7 

4 

Find  threats  to  system 

INCOSE  Pg4.17 

TP  7 

4 

Find  evidence  of  de^gn  errors 

INCOSE  Pg  4.17 

TP  7 

3 

Obtain  design  feedback 

INCOSE  Pg4.l7 

80% 

TP  7 

4 

Analyze  for  corrective  actions,  if  anomokes  detected 

INCOSE  Pg4.l7 

TP  7 

3 

Determine  corrective  actions 

INCOSE  Pg4.17 

100% 

TP-7 

4 

Ensure  interfaces  operational  when  older  component  replaced  with 
newer  component 

INCOSE  Pg  2.S 
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TP-7 

3 

Determine  system  meets  stakeholder  r  eeds 

Buede  Pg  SI 

BO% 

TP  7 

4 

Utilize  results  to  forcast  success  to  meeting  expectations  of  users 

INCOSE  Pg4.17 

TP  7 

3 

Document  va  Oat  on  results 

(Student  Derived) 

100% 

TP  7 

3 

Provide  output:  validated  system 

Buede  Pg  42 

80% 

TP  2 

1 

TP-8  (Transition) 

o%  100% 

TP  41 

2 

Identify  Transition  Opportunities 

INCOSE  Pg  3.2 

70%  100% 

T  PH 

3 

Evaluate  decagon  gates 

INCOSE  Pg  3 2 

100% 

TP  8 

3 

Determine  rf  deliverable  fulfils  business  case/rr  ssion 

INCOSE  Pg  3.2 

100% 

TP  8 

3 

Determine  f  affordable 

INCOSE  Pg  3 2 

100% 

TP  a 

3 

Determine  if  deliverable  can  be  delivered  when  needed 

INCOSE  Pg  3.2 

100% 

Tpa 

3 

Identify  required  smuation  data  for  each  activity  i  D  DAr  A  RlQJIRi  D  BY 
EACH  TRANSITION  OPPORTUNITY) 

Buede  Pg  314 

70% 

Tpa 

2 

Qualify  Production  Item 

Buede  Pg  314 

0%  0% 

tp  a 

3 

Refine  concept  for  qua  fication  of  Production  Items 

Bucde  Pg  314 

0% 

tp  a 

3 

Refine  requirements  for  qualification  of  Production  Items 

Buedr  Pg  314 

0% 

tp  a 

4 

Identify  quakficatnn  system  obf edrves 

Buede  Pg  314 

tp  a 

4 

Identify  pass/fa> .  thresholds 

Buede  Pg  314 

TPa 

4 

Develop  detailed  derrved  quakficaton  requirements 

Buede  Pg  314 

tp  a 

3 

Refine  functional  architecture  for  qualification 

Buede  Pg  314 

0% 

Tpa 

4 

Develop  functional  architectures  for  qualification  components 

Buede  Pg  314 

TPa 

4 

Allocate  derrved  requirements  to  functional  arch rt echoes 

Buede  Pg  314 

Tpa 

4 

Define  qualification  generic  physical  architecture 

Buede  Pg  314 

Tpa 

4 

Allocate  functions  to  physical  architectures 

Buede  Pg  314 

Tpa 

3 

Create  master  qualification  plan 

Buede  Pg  314 

0% 

TPa 

4 

Write  activity  level  qualification  plans  for  each  qualification  component 

Buede  Pg  314 

Tpa 

3 

Assgn  qua  fication  response  ikies 

Buede  Pg  314 

0% 

tp  a 

4 

Define  qualification  resources 

Buede  Pg  314 

tp  a 

Allocate  qualification  activities  to  resources 

Buede  Pg  314 

ipa 

4 

Define  Qualifcabon  organizations 

buede  Pg  314 

tp  a 

S 

Assign  qualification  activities  to  organizations 

Buede  Pg  314 

Tpa 

3 

Develop  qualification  schedule,  consistent  with  deveopment  schedule 

Buede  Pg  314 

0% 

tp  a 

3 

Perform  qualification  testing  and  analysis 

(Student  Delved) 

0% 

tp  a 

2 

Execute  Transition 

INCOSE  4-  IS 

§0%  100% 

tp  a 

3 

Receive  authority  to  proceed 

INCOSE  Pg  3.3 

Transition  Readness 

Assessment 

100% 

Tpa 

3 

Receive  acceptance  of  protect  Producton  Item 

INCOSE  Pg  3.3 

100% 

Tpa 

3 

Install  at  user  site 

DAG  Sec  4 .2.4.2.; 
INCOSE  Pg  4. IS 

100% 

Tpa 

4 

Train  system  users 

INCOSE  Pg  4.15 

tp  a 

4 

Affirm  users  have  knowledge  and  skills  necessary  to  perform  operation 
and  marrtenance  activities 

INCOSE  Pg4.1S 

tp  a 

3 

Confirm  system  meets  users  reeds 

INCOSE  Pg  4. IS 

100% 

tp  a 

3 

Determine  system  acceptability 

INCOSE  Pg  4.15 

BO% 

Tpa 

3 

Determine  corrective  actions  for  Production  Item  if  discrepancies  from 
acceptance  criteria 

INCOSE  Pg4.1S 

100% 

Tpa 

4 

Document  post  implementation  problems  that  may  lead  to  corrective 
actons  or  design  changes 

INCOSE  Pg  4.16 

tp  a 

3 

L'tUze  validated  system 

Buede  Pg  42 

100% 

Tpa 

3 

Provide  output:  validated  operational  concept 

Buede  Pg  42 

100% 

TMP  1 

1 

TMP-1  (Decision  Analysis) 

100%  100% 

TMP  1 

2 

Identify  Strategy  for  Making  Decision 

INCOSE  Pg  S.H 

100%  100% 

TMP  1 

3 

Formicate  problems  that  require  decisions 

INCOSE  Pg  7  J 

Utility  Theory, 
nf.ucncc  Diagrams. 
Decision  Tree 

100% 

'VP  1 

4 

Frame  problem  context 

INCOSE  Pg  7.S 

TMP  1 

S 

Utilize  h  story  of  prior  decisions 

INCOSE  Pg  S  J 

TMP  1 

s 

Utilize  assessments  from  all  relevant  persons 

INCOSE  Pg  S.H 

TMP  1 

4 

Frame  problem  scope 

INCOSE  Pg  7.S 

TMP  1 

s 

Understand  bus  ness  situation 

INCOSE  Pg  7.3 
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TMP  1 

S 

Deoomoo&e  problem  into  smaler  more  manageable  problems 

INCCJSE  Pg  7  J 

TMP  1 

s 

Identify  experts 

INCCJSE  Pg  7.3 

TMP  1 

s 

Identify  modeling  &  simulations 

DAG  Sec  4.2.3-1. 

TMP  1 

s 

Identify  supportably 

DAG  Sec  4.2.3  1. 

TMP  1 

s 

Identrfy  level  of  repair 

DAG  Sec  4.2.3  1. 

TMP  1 

Identrfy  post  fielding;  support 

DAG  Sec  4.2.3  1. 

TMP  1 

Identrfy  repair  vs  discard 

OAG  Sec  4.2.3  1. 

TMP  1 

Identrfy  cost  parameters 

□AG  Sec  4.2.3. 1. 

TMP  1 

Augment  with  prototypes 

DAG  Sec  4.2.3. 1. 

TMP  1 

Identrfy  transportability  requirements 

DAG  Sec  4.2.3 -1. 

TMP  1 

Identrfy  mamtenancc  concept 

OAG  Sec  4.2.31. 

TMP  1 

s 

Determine  affordability 

OAG  Sec  4.2.3-1. 

TMP  1 

s 

Determine  reliability 

OAG  Sec  4.2.3  1. 

TMP  1 

s 

Determine  avalability 

DAG  Sec  4.2.3  1. 

TMP  1 

s 

Determine  schedule 

DAG  Sec  4.2.3  1. 

TMP  1 

Determine  maintainability  goals 

DAG  Sec  4.2.3  1. 

TMP  1 

4 

Frame  problem  constraints 

INCCJSE  Pg  7.S 

TMP  1 

Identrfy  interoperability  constraints 

DAG  Sec  4. 2.3.1. 

TMP  1 

3 

Identify  project  decision 

INCCJSE  Pg  S.B 

100% 

TMP  1 

4 

Identrfy  all  personnel  with  knowledge  and  expeneoce  relevant  to 
decision 

INCCJSE  Pg  S Ji 

TMP  1 

4 

Util  ire  imcetamty  as  catalyst  of  future  performance 

INCCJSE  Pg  7.3 

TMP  1 

4 

Utilize  systemic  thnking  to  connect  current  to  future  situations 

INCCJSE  Pg  7.3 

TMP  1 

4 

Utilize  daiog  to  foster  learning; 

INCCJSE  Pg  7.3 

TMP  1 

4 

Utilize  daiog  to  danfy  actions 

INCCJSE  Pg  7.3 

TMP  1 

3 

Communicate  decisions  with  stakeholders 

INCCJSE  Pg  7.S 

100% 

4 

Invoke  necessary  disciplines  in  decision  approval 

INCCJSE  Pg  72 

TMP  1 

4 

Invoke  necessary  stakeholders  in  decision  aporoval 

INCCJSE  Pg  72 

TMP  1 

3 

Establish  clear  obiectrves  for  project  decisions 

INCCJSE  Pg  7  J 

100% 

TMP  1 

4 

Determine  entry/ exit  criteria  for  decs  ion 

INCCJSE  Pg  7.1 

TMP  1 

4 

Identrfy  decision  purpose 

INCCJSE  Pg  7.2 

IMP  i 

4 

Identify  decision  support  activities 

INCCJSE  Pg  72 

TMP  1 

S 

Identrfy  decision  chairperson 

INCCJSE  Pg  72 

TMP  1 

s 

Identrfy  decision  attendees 

INCCJSE  Pg  7.2 

TMP  1 

Identrfy  decision  location 

INCCJSE  Pg  72 

TMP  1 

Identrfy  decision  agenda 

INCCJSE  Pg  72 

TMP  1 

Identrfy  decision  method  of  conduct 

INCCJSE  Pg  72 

TMP  1 

Identify  decision  evidence  to  be  evaluated 

INCCJSE  Pg  72 

TMP  1 

Identify  decision  actions 

INCCJSE  Pg  72 

TMP  1 

s 

Identify  decision  method  for  dosing  review 

INCCJSE  Pg  7.2 

TMP  1 

2 

Execute  Decision  Making  Strategy 

DAG  4 .2 .3.1 

100%  100% 

V-  . 

3 

Select  decision  criteria 

DAG  Sec  4. 2. 3.1.; 
INCCJSE  Pg  7.1 

House  of  Quality. 

Utility  Theory, 
nf  uence  Diagrams. 
Decision  Tree. 
Sensitivity  Analysis 

100% 

TMP  1 

4 

Define  evaluation  criteria 

INCCJSE  Pg  7.S 

TMP  1 

S 

Select  criteria  that  arc  measurable 

B-uede  Pg  149.  iNCOSE 

TMP  1 

Express  criteria  in  understood  units 

Buede  Pg  149 

TMP  1 

s 

Select  criteria  with  demonstrable  links  to  customer  needs  and 

system  requirements 

Suede  Pg  14£ 

TMP  1 

s 

Maintain  need  based  balance  among  often  conflicting  criteria 

Buede  Pg  14U 

TMP  1 

4 

Define  and  assgjn  value  we  grits  to  criteria 

Buede  Pg  36S,  tNCOSE 
P*7S 

TMP  1 

4 

Coordnatc  criteria/ value  weights  with  stakeholders 

INCCJSE  Pg  7.6 

TMP  1 

3 

Identify  trade  studies 

DAG  Sec  4.2.3  1.; 
INCCJSE  Pg  7 .5 

100% 

TMP  1 

4 

Establish  dear  trade  offs 

OAG  Sec  4.2.3. 1.; 
INCCJSE  Pg  7.3 

TMP  1 

4 

Develop  creative  and  unque  alternatives 

INCCJSE  Pg  7.3 

TMP  1 

S 

Define  alternatives 

INCCJSE  Pg  7.S 

TMP  1 

S 

Use  value  creation  lens  for  developing  opportunities 

INCCJSE  Pg  7  J 

TMP  1 

s 

Determine  rf  requirements  can  be  traded  against  constraints 

INCCJSE  Pg  7.6 

TMP  1 

s 

Use  trade  offs  to  show  performance  variations 

Buede  Pg  149 

TMP  1 

Use  trade  offs  to  show  cost  variations 

Buede  Pg  149 

103 


TMP  1 

Use  trade  offs  to  show  schedule  variations 

Suede  Pg  149 

TMP  1 

Use  trade  offs  to  show  risk  impact  variations 

Buede  Pg  149 

TMP  1 

S 

Determine  if  architecture  features  can  be  traded  agamst  dictated 
cqupment 

INCOSE  Pg  7.6 

TMP  1 

s 

Determine  if  architecture  features  can  be  traded  against  interface 
reqisrements 

INCOSE  Pg  7.6 

TMP  1 

s 

Determine  if  alternative  functional  cnoces  can  be  traded  to 
determine  an  optimal  configuration 

INCOSE  Pg  7.6 

TMP  1 

s 

Determine  if  alternative  performance  choices  can  be  traded  to 
determine  an  optimal  configuration 

INCOSE  Pg  7.6 

TMP  1 

4 

Select  analysis  methods 

DAG  Sec  4.2X1.; 
INCOSE  Pg  7  J 

TMP  1 

s 

Use  smulatcn  and  experimental  design  to  perform  trade  offs 

Buede  Pg  149 

TMP  1 

4 

Define  measures  of  merit 

INCOSE  Pg  7.S 

TMP  1 

A 

Select  canddates  for  study 

INCOSE  Pg  73 

TMP  1 

3 

Evaluate  alternatives 

INCOSE  Pg  7.S 

House  of  Quafcty. 

Utility  Theory, 
inf  uence  Diagrams. 
Oeasaon  Tree, 
Sensitivity  Analysis 

100% 

TMP  1 

A 

Determine  probability  of  alternatives 

Buede  Pg  361 

TMP  1 

A 

Determine  order  of  preferences 

Buede  Pg  361 

TMP  1 

A 

Determine  acceptable  substitutions 

Buede  Pg  361,  INCOSE 
PgS-8 

. 

Use  value  creation  lens  for  evaluating  opportimities 

INCOSE  Pg  7.3 

TMP  1 

A 

Analyze  results 

INCOSE  Pg  7.S 

TMP  1 

S 

Gather  meanngfii  and  reliable  data 

INCOSE  Pg  7.3 

TMP  1 

Make  value  judgements  of  trade  offs  by  customer 

Buede  Pg  149 

TMP  1 

Evaluate  consequences  of  alternative  choices 

INCOSE  Pg  S3 

TMP  1 

Avoid  analysis  paralysis 

INCOSE  Pg  7.3 

TMP  1 

A 

Review  results  with  stakeholders 

INCOSE  Pg  7.S 

IMP-1 

S 

Amow  customer  to  modify  requirements  based  on  trade-offs 

buede  Pg  149 

TMP  1 

S 

AAow  customer  to  participate  in  dcvclopng  solution  based  on  trade 
offs 

Buede  Pg  149,  NCOSE 
Pg  7.2 

TMP  1 

A 

Investigate  consequences  of  implementation 

INCOSE  Pg  7.S 

TMP  1 

A 

Utilize  scenario  planning  to  test  assumptions  about  future 

INCOSE  Pg  7.S 

TMP  1 

A 

Select  best  alternative 

Buede  Pg  361,  INCOSE 
PgS.8  INCOSE  Pg  7.S 

TMP  1 

S 

Delay  commitment  to  last  moment 

INCO&t  Pr  7.4 

TMP  1 

4 

Communicate  new  directions  from  decision 

INCOSE  Pg  S.8 

TMP  1 

3 

Document  decision 

INCOSE  pg  s.a 

100% 

TMP  1 

4 

Document  approved  decision 

INCOSE  pg  s.a 

TMP  1 

4 

Document  relevant  data  and  support ng  documentation 

INCOSE  pg  s.a 

TMP  1 

S 

Document  rationale 

INCOSE  pg  s.a 

TMP  1 

S 

Document  assumptions 

INCOSE  Pg  S .8 

TMP  1 

s 

Document  constraints 

INCOSE  Pg  s.a 

TMP  1 

s 

Document  support  ng  analysis 

INCOSE  Pg  s.a 

TMP  1 

4 

Maintain  history  of  prior  studies  and  decisions  in  case  old  questions 
reappear 

INCOSE  Pg  S A 

Oeasaon  Analysis 
Record 

TMP  2 

1 

TMP-2  (Technical  Planning) 

0%  100% 

TMP  2 

2 

Plan  Systems  Engineering 

DAG  Sec  4.2.3  2. 

<7*  100% 

TMP  2 

3 

Collaborate  wtth  protect  managers  to  develoo  protect  plan 

INCOSE  Pg8.ll 

SE  Plan 

10 0% 

TMP  2 

4 

Consider  where  post  experiences  and  intuition  have  been  a  handicap,  for 
architecture  development 

INCOSE  Pg8.3 

TMP  2 

4 

Consider  developing  architectural  alternatives  to  meet  stakeholder 
requirements 

INCOSE  Pg  8.2 

TMP  2 

3 

Address  scope  of  technical  effort 

DAG  Sec  4.2.3 .2. 

100% 

TMP  2 

3 

Identify  people 

DAG  Sec  4.2.43. 

integrated  Product 
Team 

100% 

TMP  2 

4 

Use  Integrated  Product  Team  for  analysts 

INCOSE  Pg  4.8 

TMP  2 

3 

Identify  processes 

DAG  Sec  4.2.43. 

100% 

TMP  2 

4 

Determine  program  decagon  process 

DAG  Sec  4.2.43. 

TMP  2 

3 

Develop  integration  plan 

INCOSE  Pg4.12 

80% 

TMP  2 

3 

Identify  critical  producaoility  requirements 

INCOSE  Pg  8.4 

80% 
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TMP2 

3 

Establish  contractual  context  and  constraints 

IhCQSE  Pg  8.1 

100% 

TMP  2 

4 

Examine  mptementabon  (TP  4)  for  constraints 

DAG  Sec  4.2.3 2. 

TMP  2 

4 

Examine  integration  (TP  S)  for  constrants 

DAG  bee  4.2.3 -2. 

TMP  2 

4 

Examine  verification  (TP  6)  for  constraints 

DAG  Sec  4  2.3-2. 

TMP  2 

4 

Examine  vahdabon  (TP  7)  for  constraints 

DAG  Sec  4  2.3.2. 

TMP  2 

4 

Examine  transition  (TP  8)  for  constraints 

DAG  Sec  4.2.3 .2. 

TMP  2 

3 

Assist  protect  manager  with  contract  negotiations 

IhCQSE  PgS.l 

100% 

TMP  2 

3 

Address  conformance  with  society  expectations 

IhCQSE  Pg  4.1 

100% 

TMP  2 

3 

Address  conformance  with  legis-ativc  requ  'ements 

INCQSE  Pg4.1 

100% 

TMP  2 

3 

Establish  quality  management 

INCOSE  ®g  8.12 

80% 

TMP  2 

3 

Develop  plan  to  qua:  fy  suppliers 

DAG  Sec  4.2.4  4. 

100% 

TMP  2 

3 

Develop  raceMng  nspection  plan 

DAG  Sec  4.2.4  4 

100% 

TMP  2 

3 

Develop  manufacturng  system 

DAG  Sec  4.2.4  4. 

0% 

TMP  2 

4 

Develop  manufacture  processes 

DAG  Sec  4.2.4  4. 

TMP  2 

4 

Develop  manufacture  procedures 

DAG  Sec  4.2.4  4. 

TMP  2 

4 

Identify  packag  ig 

DAG  Sec  4.2.4  4. 

TMP  2 

4 

Identify  handling 

DAG  Sec  4.2.4  4. 

TMP  2 

4 

Identify  storage 

DAG  Sec  4.2.4  4. 

TMP  2 

4 

Identify  equipment  mamtainance  program 

DAG  Sec  4.2.4  4. 

TMP  2 

Identify  criteria  for  refurt)  inspection 

DAG  Sec  4.2.4  4. 

TMP  2 

3 

Develop  configuration  management  plan 

INCOSE  Pg  8.S 

100% 

TMP  2 

3 

Develop  information  management  plan 

INCQSE  Pg  8.b 

100% 

TMP  2 

3 

Define  test  plans  and  schcdjlc 

Buede  Pg  314 

100% 

TMP  2 

4 

Develop  verification  schedule,  consistent  wttn  development  schedule 

Buede  Pg  314 

TMP  2 

4 

Schedule  verification  enabkng  systems 

INCOti  Pg4.14 

TMP  2 

4 

Define  analysis  schedule 

Buede  Pg  314 

TMP  2 

3 

Prepare  for  verification 

DAG  Sec  4.2.4  4.; 
IhCQSE  Pg  8.15 

100% 

TMP  2 

4 

Identify  tester  catibraton  p-ogram 

DAG  Sec  4.2.4  4. 

TMP  2 

4 

Define  verification  organizations 

Buede  Pg  314 

TMP  2 

4 

Define  verification  resources 

Buede  Pg  314 

TMP  2 

3 

Prepare  for  validation 

DAG  Sec  AJ.AA.. 
INCQSE  Pg  8.1S 

100% 

TMP  2 

4 

Define  validation  resources 

Buede  Pg  314 

TMP  2 

4 

Define  validation  organizations 

Buede  Pg  314 

TMP  2 

3 

Prepare  transition  strategy 

IhCQSE  Pg  4.1S 

100% 

TMP  2 

4 

Assess  operational  environment 

IhCQSE  Pg4.1S 

TMP  2 

4 

Prepare  operator  trauvng  strategy 

IhCQSE  Pg4.1S 

TMP  2 

4 

Prepare  logistics  support  strategy 

IhCQSE  9g  4. IS 

TMP  2 

4 

Develop  installation  procedure 

IhCQSE  Pg4.lS 

TMP  2 

4 

Define  operational  concept  for  qualification  of  dc  verabes 

Buede  Pg  314 

TMP  2 

4 

Define  requirements  for  quakftcation  of  deliverables 

Buede  Pg  314 

TMP  2 

4 

Define  functional  architecture  for  qualification 

Buede  Pg  314 

TMP  2 

2 

Implement  Technical  Plan 

INCQSE  PgS.l  13 

ntegrated  Master  Plan 

50%  100% 

TMP  2 

3 

Utilize  SE  Plan  (SEP)  and  ntegrated  Master  Schedule  (MSI 

IhCQSE  Pg  8.11 

100% 

TMP  2 

3 

Develop  system  architecture 

IhCQSE  Pg  8.2 

Do  DAE  System  View 

80% 

TMP  2 

3 

PartKipatc  in  process  compliance  revevrs 

IhCQSE  Pg  8.12 

100% 

TMP  2 

3 

Perform  tectincal  management  actfMbes  (planning,  schedidrvg,  reviewing, 
and  auditing  to  Si  process) 

INCOSEPg8.ll 

100% 

TMP  2 

3 

Nrfanw pn  ■  trada  Mdta on  :c,t.  giaMg ndtKUol 

performance 

INCQSE  Pg  8.1 

100% 

TMP  2 

3 

Perform  interoperabdJty  analysts 

INCQSE  Pg  8.3 

80% 

TMP  2 

3 

Perform  manufacturng  and  producbibty  analysis 

INCQSE  Pg  8.4 

80% 

TMP  2 

3 

Perform  cost  effcetweness  analysts 

INCOSE  Pg  8.8 

SO% 

TMP  2 

3 

Perform  life  cycle  cost  analysis 

INCQSE  Pg  8.9 

100% 

TMP  2 

3 

Maintain  resource  management 

INCQSE  Pg  8.13 

100% 

'MP2 

2 

Evaluate  Plan  to  Address  Needs 

INCQSE  Pg  3  A  9 

0%  0% 

TMP  2 

3 

Assess  impacts  of  product  modficatons  djrmg  production 

INCOSE  Pg  3.8 

0% 

TMP  2 

3 

Assess  impacts  of  product  modficatons  during  usage 

INCOSE  Pg  3  J 

0% 

TMP  2 

3 

Assess  impacts  of  product  maintenance  modifications  during  usage 

IhCQSE  Pg  3.9 

0% 

TMP  3 

1 

TMP  3  (Technical  Assessment) 

100%  100% 

*MP  3 

2 

Prepare  for  Technical  Assessment 

DAG  Sec  4  2  3-3. 

100%  100% 

TMP  3 

3 

Develop  review  process 

DAG  Sec  4.2.3  J. 

100% 

TMP  3 

4 

OevcJop/utdize  protect  plans 

INCQSE  Pg  S.S 
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TMP  3 

4 

'What  gets  rreasu'cd  gets,  done* 

INCOSE  Pg  S.5 

TMP  3 

A 

Utilize  prior  protect  status 

INCOSE  Pg  S.S 

TMP  3 

A 

Utilize  key  personnel 

INCOSE  Pg  S.S 

TMP  3 

A 

Utilize  key  resources 

INCOSE  Pg  S.S 

TMP  3 

3 

Determine  technical  performance  measures 

INCOSE  PgS.S 

*pMs 

100* 

TMP  3 

A 

Determine  protect  performance  measures 

INCOSE  PgS.S 

TMP  3 

3 

Schedule  frequent  assessments 

INCOSE  Pg  S.b 

IPT  Meetings 

IDO* 

rMP  3 

2 

Perform  Technical  Assessment 

INCOSE  PgS.S 

100*  100* 

TMP  3 

3 

Gain  insight  into  contractor  processes 

DAG  Sec  4.2.4  4. 

ICC* 

TMP  3 

3 

Monitor  critical  tasks 

INCOSE  PgS.S 

integrated  Master 
Schedule 

IDO* 

TMP  3 

3 

Monitor  new  technologies 

INCOSE  Pg  S.S 

10 o* 

TMP  3 

3 

Conduct  technical  reviews 

DAG  Sec  4.2  J.3.; 
INCOSE  Pg  S.S 

10O* 

TMP  3 

4 

Evaluate  hardware  soJutons 

DAG  Sec  4.2.4  4. 

TMP  3 

4 

Evaluate  manufacturing  solutions 

DAG  Sec  4.2.4  4. 

TMP  3 

4 

Evaluate  software  solutions 

DAG  Sec  4.2.4  4. 

TMP  3 

4 

Evaluate  test  resists 

DAG  Sec  4.2.4  4. 

TMP  3 

4 

Analyze  trade  studies 

Bjede  Pg  2b8 

TMP  3 

4 

Assess  Mrties* 

TMP  3 

Address  reliability 

Bjede  Pg  2b  7 

TMP  3 

S 

Address  avaUaOikty 

Bjede  Pg  2b 7,  INCOSE 

-• 

TMP  3 

S 

Address  maintainability 

Bjede  Pg  2b  7,  WCOSE 
PR  3.9 

TMP  3 

Address  usability 

Bjede  Pg  2b  7 

TMP  3 

Address  supportabirty 

Bjede  Pg  2b  7 

VI  : 

Address  durability 

Bjcde  Pg  2b  7 

TMP  3 

s 

Address  affordability 

Bjede  Pg  2b  7,  INCOSE 
PgS-S 

TMP  3 

4 

Assess  concurrent  engneenrg 

IMP  3 

s 

Address  concurrent  engineer ir*  issues  to  unpact  on  manufacturing 

Bjede  Pg  2b  7,  NCOSE 
P|3J 

TMP  3 

s 

Address  concurrent  engineering  issues  to  impact  on  deployment 

Bjede  Pg  2b7 

TMP  3 

Address  concurrent  engineering  issues  to  impact  on  training 

Bjede  Pg  2b  7 

TMP  3 

s 

Address  concurrent  engineering  issues  to  impact  on  dsposal 

Bjede  Pg  2b  7,  INCOSE 
Pg  3.9 

TMP  3 

4 

Assess  product  modifications 

INCOSE  Pg  3.8 

TMP  3 

4 

Assess  resources 

TMP  3 

4 

Evaluate  completion  of  required  accomplishments 

DAG  Sec  4.2.3  J.; 
INCOSE  Pg  S.S 

TMP  3 

4 

Determine  deviations  in  project  quality 

INCOSE  Pg  S.S 

TMP  3 

3 

Conduct  peer  reviews 

INCOSE  pg4.10 

100* 

TMP  3 

3 

Evaluate  to  exit  criteria 

DAG  Sec  4.2.3  J.; 
INCOSE  Pg  S.S 

IDO* 

TMP  3 

4 

Determine  actual  and  projected  cost  against  budget 

INCOSE  PgS.S 

TMP  3 

4 

Determine  actual  and  projected  time  against  schcdide 

INCOSE  PgS.S 

TMP  3 

4 

Evaluate  effectiveness  of  personnel 

INCOSE  Pg  S.S 

TMP  3 

4 

Evaluate  adequacy  of  project  infrastructure 

INCOSE  Pg  S.S 

TMP  3 

4 

Evaluate  availability  of  project  infrastructure 

INCOSE  PgS.S 

TMP  3 

4 

Evaluate  project  progress  against  established  criteria 

INCOSE  PgS.S 

TMP  3 

4 

Evaluate  project  progress  against  established  milestones 

INCOSE  Pg  S.S 

TMP  3 

4 

Conduct  reviews  to  determine  readiness  to  proceed  to  next  milestone 

INCOSE  PgS.S 

TMP  3 

3 

follow  through  with  results 

100* 

TMP  3 

4 

Determine  recommendations  for  adjustments  to  project  plans 

INCOSE  Pg  S.S 

TMP  3 

s 

Determine  corrective  actions  for  deficiencies 

DAG  Sec  4.2.3  J., 
INCOSE  Pg  S.S 

TMP  3 

s 

Make  recommendations  for  adjustments  to  project  plans 

INCOSE  Pg  S.S 

TMP  3 

4 

Make  decision  to  procceed  or  not  to  proceed,  when  assessment 
supports  tollgate 

INCOSE  PgS.6 

TMP  3 

4 

Generate  change  requests 

INCOSE  Pg  S.S 

TMP  3 

S 

Establish  changes  to  schedule  to  reflect  actions  taken 

INCOSE  Pg  S.b 

TMP  3 

s 

Establish  work  items  to  reflect  actions  taken 

INCOSE  Pg  S.b 

TMP  3 

4 

Implement  corrective  actons  for  deficiencies 

DAG  Sec  4.23.3.; 
INCOSE  Pg  S.b 

TMP  3 

s 

Initiate  correct rve  actions  when  assessment  ndcates  deviation  from 
approved  plan 

INCOSE  Pg  S.b 

TMP  3 

s 

Initiate  preventive  actions  when  assessment  indicates  trend  toward 
deviation 

INCOSE  Pg  S.6 
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TMP3 

s 

Initiate  problem  resolution  when  assessment  indicates  non 
conformance  wrth  performance  success  criteria 

INC05E  Pg  S.6 

vr  : 

4 

Compile  project  performance  measures  assessment 

INCOSE  PgS.S 

TMP  3 

4 

Identify  risk  assessment  situations 

INCOSE  Pg  S.S 

TMP  3 

4 

Develop  effective  feedback  control  system 

INCOSE  Pg  S.6 

TMP  3 

4 

Communicate  status  per  agreements  and  pokcies 

INCOSE  pg  S.S 

*MP4 

1 

T MP4  (Requirements  Management) 

10%  10O% 

TMP  4 

2 

Buede  Pg  129 

100*  100% 

TMP  4 

3 

Identify  stakeholders  (customers,  end-users,  regulatory  agencies, 
representation  for  future  generations,  etc.) 

INCOSE  Pg  7.7 

MCOSE  Toots 

Database  Working 
Group  Database 
(INCOSE  Pg  7.7) 

100% 

TMP  4 

4 

Identify  who  has  right  to  have  operational  requirement 

buede  Pg  129 

TMP  4 

3 

Establish  means  for  stakeholder  interaction 

Buede  Pg  129 

100% 

TMP  4 

2 

Define  System  Capabilities  and  Performance  Objectives 

INCOSE  Pg  7.6-12 

30%  100% 

TMP  4 

3 

Understand  needs  of  stakeholders  to  support  architecture  design  process 

INCOSE  Pg  7.10 

dertifv  capabiity  gaps 
&  threaten  ng  systems 

100% 

TMP  4 

3 

Elicit  and  capture  requirements 

INCOSE  Pg  7.6 

OoDAFOV  1. 
interviews.  Focus 
Groups,  Delphi 
Technique 

100% 

TMP  4 

4 

Collect  requirements  from  ad  communications  and  negotiations 

INCOSt  Pg  7.6 

TMP  4 

4 

Capture  source  for  each  requirement 

INCOSE  Pg  7.6 

TMP  4 

4 

Capture  rationale  for  each  requirement 

INCOSE  Pg  4.4 

TMP  4 

3 

Address  non  functional  requirements  (iktics)  early 

INCOSE  Pg  7.9 

30% 

TMP  4 

3 

Develop  teenneal  performance  measures  for  each  requirement 

INCOSE  Pg  7.12 

100% 

INCOSE  Pg  7.12 

TMP  4 

2 

Validate  Requirements  Oevdoomcnt  Process 

Suede  Pg  41 

50%  SO% 

TMP  4 

3 

Generate  operational  analysis  models  to  test  concept  and  requirements 
validity 

INCOSE  Pg  7.9 

OoDAf  Operational 

Views 

SO% 

TMP  4 

4 

Carry  analysis  to  one  level  deeper  than  seems  necessary 

Buede  Pg  1S8 

TMP  4 

4 

Carry  analysis  to  one  level  broader  than  seems  necessary 

Suede  Pg  1S8 

TMP  4 

4 

Determine  how  it  a  known  that  requirement  is  ’right' 

Buede  Pg  129 

TMP  4 

2 

Ensure  Requirements  Feasibility  and  ValKfrty 

Buede  Pg4a 

10%  100% 

TMP  4 

3 

Ensure  customer  and  consumers  understand  requirements 

Buede  Pg  1S8 

100% 

TMP  4 

4 

Identify  scenarios 

INCOSE  Pg  7 A 

intennews  with 
opertors  of  current  or 
svmlar  systems, 
potential  end  users, 
and  meetings  with 
interface  Workng 
Group 

TMP  4 

4 

Resolve  uncertain  «s  wtn  reqwrements 

INCOSE  Pg  7.11 

TMP  4 

3 

Analyze  requirements  for  clarity,  completeness,  and  consatency 

INCOSE  Pg  4.4. 

DAG  Sec  4.2.43. 

80% 

TMP  4 

4 

Analyze  requirements  against  al  commumcatons  and  negotiations 

INCQSE  Pg  7.6 

TMP  4 

4 

Analyze  impact  of  emerging  technologies 

DAG  Sec  4.2.3  4. 

TMP  4 

4 

Analyze  impact  of  current  threats 

DAG  Sec  4.2.3  4. 

TMP  4 

4 

IMhi  f-aiiure  Modes  Effects  and Crtticakty  Mgh  PMKA]  V  Marti 
Analysis  to  dentrfy  critical  system  level  requirements 

INCOSE  Pg  4.6 

TMP  4 

3 

Analyze  requirements  for  fcasib  -ty  by  interdisciplinary  team 

INCOSE  Pg  7.11 

10% 

TMP  4 

3 

Validate  requirements  at  all  levels 

INCQSE  Pg  7.9 

100% 

TMP  4 

4 

Validate  requirements  to  all  communications  and  negotiations 

INCQSE  Pg  7.6 

TMP  4 

3 

Assess  if  reqiarements  are  verifiable 

INCOSE  Pg  7.11 

70% 

TMP  4 

2 

Document  Requirements 

INCOSE  Pg  4.3,  4  4 

30%  100% 

TMP  4 

3 

Develop  concept  document  to  capture  implementation  free  understanding 
of  stakeholder's  needs 

INCOSE  Pg  7.9 

tntba 1  Capacities 

Oocument  or 
equivalent,  AFPO  10 

28 

100% 

TMP  4 

4 

Develop  concept  document  to  capture  behavioral  characteristics  of 
subsystems 

INCOSE  Pg  7.9 

OoDAFOV  2 

TMP  4 

4 

Develop  concept  document  to  capture  manner  m  which  people  wtfl 
interact  w*h  system 

INCOSE  Pg  7.9 

OoDAFOV  2 
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TMP4 

4 

Develop  concept  document  to  capture  external  system  interaction 

INC05E  Pg  7.9 

CCSI/M  31 KJ,  APPO  10 
28 

TMP4 

3 

Create  templates  for  constructing  requremerrts  statements 

INC  OS*  Pg  4. fa 

80% 

TMP  4 

3 

Write  rationale  for  each  requirement  (often  uncovers  real  requirement) 

Suede  Pg  1S8 

80% 

TMP  4 

4 

Estabfcsh  bas  s  for  requirements  to  supoort  system  over  its  life 

INCOSE  Pg  7.9 

TMP  4 

4 

EstaNtsh  basts  for  test  planning,  system  test  requirements,  and 
environmental  simulator  requirements 

INCOSE  Pg  7.9 

TMP  4 

4 

Provide  basts  for  computation  of  system  capacity 

INCOSE  Pg  7.9 

TMP4 

3 

Develop  requirements  dagran  to  capture  hierarchies,  derivation, 
satsfactuon,  and  verification  relation  snips 

INCOSE  Pg  7.8 

OoDAf  OVS.SV'S 

30% 

TMP  4 

3 

Ptace  establ  shed  requrements  under  configuration  control 

INCOSE  Pg  4.4 

60% 

TMP  4 

3 

Convey  documented  requirements  to  stakeholders 

Buede  Pg  129 

Originating 

requirements 

i«C  MHl 

Requirements  Baseline 

10 0% 

TMP  4 

2 

ensure  Traceability  of  Requirements 

DAG  Sec  4.2.3  4.; 

Suede  Pg  1S8,  INCOSE 
Pg  3.10 

100%  100% 

TMP  4 

3 

Determine  traceability  tool 

Suede  Pg  23,  INCOSE 
Pg  4.4 

initial  Capabilities 
Document.  Excel 
Spreadsheet 

100% 

TMP  4 

4 

Develop  requrements  verrficatcn  traceabiity  matnx 

INCOSE  Pg  4.13 

TMP  4 

3 

Provide  trace aoil it y  from  operational  needs  to  requrements 

INCOSE  Pg  7.9 

100% 

TMP  4 

4 

Determine  requirements  traced  to  functions 

Suede  Pg  S3,  INCOSE 
Pg4  * 

TMP  4 

4 

Determine  requirements  traced  to  components 

Suede  Pg  S3.  INCOSE 
Pg4-9 

TMP  4 

4 

Determine  requirements  traced  to  inputs/outputs  assigned  to  interfaces 

Suede  Pg  S3.  INCOSE 
Pg4.ll 

TMP  4 

4 

Enter  data  to  trace  ability  matrn 

INCOSE  Pg4.l7 

IMP4 

3 

Mamtam  traceabwry  to  requrements 

INCOSE  Pg  4.fc 

ioo% 

TMP  4 

4 

Confirm  upward  traceadility  of  requirements 

□AG  Sec  4.2.4  J. 

TMP  4 

4 

Confirm  downward  traceability  of  requirements 

DAG  Sec  4.2.4 3. 

TMP  4 

4 

ContinuaPy  verify  technical  parameters  to  mondor  trend 

INCOSE  Pg  7.12 

TMP  4 

4 

Control  external  interfaces 

Suede  Pg  1S8 

TMP  4 

4 

Control  internal  interfaces 

Suede  Pg  1S8 

TMP  4 

4 

Input  information  to  requirements  verification  traceability  matrm 

INCOSE  Pg  4.14 

TMP  4 

2 

Istabliih  Process  for  Requrements  Chances 

Suede  Pg  129 

80%  100% 

TMP  4 

3 

Establish  plan  to  correct /change  requrements 

Suede  Pg  1S8 

100% 

TMP  4 

3 

Establish  process  for  evaluating  impact  of  system  modifications 

Suede  Pg  129 

80% 

TMP  4 

3 

Document  requrements  charges 

DAG  Sec  4.2.3  4 

100% 

TMP  4 

4 

Record  rationale  for  changes 

DAG  Sec  4.2.3  4. 

TMP  4 

4 

Record  corrective  actions 

INCOSE  Pg  4.14 

TMP  4 

3 

Assess  product  modficatiorts 

INCOSE  Pg  321 

80% 

TMP  4 

4 

Record  evidence  that  system  element  satisfies  requirements,  or  not 

INCOSE  Pg  4.14 

TMP  4 

4 

Record  evidence  that  system  satisfies  requrements.  or  not 

INCOSE  Pg  4.14 

TMP  S 

1 

TMD  C  / q:.l  ij _ nn.\ 

1  rflr'J  jluaA  Rn  aallalfSCl  II  Cfl  VJ 

40%  100% 

.... 

2 

Risk  Ptanrwig 

DAG  Sec  4 23  <> 

Risk  Management 
framework 

0% 

TMP  S 

3 

Establish  nsk  management  process 

DAG  Sec4.2.3_S.; 

Suede  Pg  382,  INCOSE 
PgS.ll 

Reference  Profect 
Management  Institute, 
and  'Risk  Management 
Standard’  generated 
by  institute  of  Rsk 
Management 

100% 

TMP  S 

3 

Coordinate  nsk  management  activities 

DAG  Sec  4.2.3.S.; 
INCOSE  Pg  3.4 

50% 

TMP  S 

4 

Identify  what  shoidd  be  done 

DAG  Sec  4.2.3 -S. 

TMP  S 

4 

Estabfcsh  schedule 

DAG  Sec  4.2.3.S. 

integrated  Master 
Schedule 

TMP  S 

4 

Assign  responsibilities 

DAG  Sec  4.2.3-S. 

TMP  S 

3 

U  blue  project  nsk  assessments 

INCOSE  Pg  S.10 

100% 
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TMP  S 

4 

Utilize  pnor  rak  matrix 

INCOSE  Pg  S.10 

Examine  successes, 
failures,  problems,  and 

solutions  of  simkar 

pnor  prefects, 
expected  value  model 
(INCOSE  Pg  7.1S) 

TMP  S 

4 

Utilize  project  plans  and  performance 

INCOSE  Pg  S.10 

IMPS 

2 

Risk  Identification 

DAG  Sec  4.2.154 

Buede  Pg  314,  MCOSE 
PgS.iaS.il 

Documentation 
Reviews,  information 
-r  t'ir-r  R| 

(Bramstnrming.  Delphi 
'ecfviique.  Interviews. 
SWOT  (Strength 
Weakness  Opportunity 
’hreat)  Analyse) 

50%  100% 

TMP-S 

2 

Identify  cost  risk 

DAG  Sec  4.2.3-S.; 
INCOSE  Pg  7.14 

Checklists 

100% 

TMP  S 

a 

Identify  schedule  nsk 

□AG  Sec  4.2.3-S.; 
INCOSE  Pg  7.14 

ntegrated  Master 
Schedule,  Critical  Path 
Analysis,  Assumption 
and  Constraint  Analyse 

100% 

TMP  S 

a 

Identify  performance  nsk 

DAG  Sec  4.2.3.S.; 
INCOSE  Pg  7.14 

_  m 

techniques  (Cause 
Effect  Diagrams,  fault 
Event  Trees, 

System,/ Process  flow 
Charts,  Influence 

100% 

IMPS 

a 

Identify  technology  maturity  risks 

DAG  Sec  4.2.3-S. 

Graph  of  planned  va\ze 
of  key  parameter 
platted  against 
calendar  time  0  NCOS! 
Pg  7.17) 

100% 

TMP  S 

4 

Identify  root  cause  for  technology  maturity  risks 

DAG  Sec  4.2.3-S. 

TMP  S 

a 

Identify  suppler  capaoilty  risks 

DAG  Sec  4.2.3-S. 

80% 

TMP  S 

4 

Identify  root  cause  for  supplier  capaOikty  risks 

DAG  Sec  4.2.3-S. 

TMP  S 

a 

Identify  design  maturation  risks 

DAG  Sec  4.2.3  S. 

S0% 

TMP  S 

4 

Identify  root  cause  for  design  maturation  nsks 

DAG  Sec  4.2.3-S. 

a 

Review  polenta!  shortfall  against  expectations 

DAG  Sec  4.2.3-S. 

100% 

V- 

2 

Risk  Analysis  (Qualitative  &  Quantitative) 

DAG  Sec  4.2.3.S.; 

Buede  Pg  382 

ARENA.  COPi.MA'LAB 
State  flow  Modeler, 
Crystal  Ball  (Excel  add 
m) 

100%  100% 

TMP  S 

a 

Identify  Areas  for  Analysis 

100% 

TMP  S 

4 

Analyze  requirements 

INCOSE  Pg  7.18 

TMP  S 

4 

Analyze  current  /  proposed  staff  ng 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Analyze  current  /  proposed  process 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Analyze  current  /  proposed  design 

DAG  Sec  4.2.3 -S. 

TMP  S 

4 

Analyze  current  /  proposed  supplier 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Analyze  current  /  proposed  operational  employment 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Analyze  current  /  proposed  resources 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Analyze  current  /  proposed  depend anaes 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Analyze  negative  trends 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Examine  wide  range  of  operational  scenarios 

Buede  Pg  2b  7 

TMP  S 

a 

Assess  ProbabUrty 

INCOSE  Pg  S.10 

100% 

TMP  S 

4 

Identify  probability  of  threats 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Identify  probability  of  technology  maturity  nsks 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Identify  probabkty  of  supplier  capability  nsks 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Identify  probability  of  design  maturation  nsks 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Identify  probability  of  performance  vs  plan  risks 

DAG  Sec  4.2.3-S. 

TMP  S 

a 

Assess  Consequence 

INCOSE  Pg  S.10 

100% 

TMP  S 

4 

Identify  consequence  of  threats 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Identify  consequence  of  technology  maturity  nsks 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Identify  consequence  of  supplier  capability  nsks 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Identify  consequence  of  design  maturation  nsks 

DAG  Sec  4.2.3-S. 

TMP  S 

4 

Identify  consequence  of  performance  vs  plan  risks 

DAG  Sec  4.2.3-S. 
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TMPS 

2 

Risk  Handl**; 

DAG  Sec  4.2.33.; 
INCOSE  PgS.ll 

100%  ioo% 

TMP  S 

2 

Establish  risk  handling  approaches  for  moderate  and  hgn  nsk  items 

(NCOS*  Pg  7.17 

Risk  Management  Pan 

100% 

IMPS 

4 

ffrsk  Mktigation 

DAG  Sec  4.233.; 

Bucde  Pg.  314 

TMP  S 

S 

Determine  planning  for  risk  mitigation 

DAG  Sec  4.2.33. 

TMPS 

S 

Determine  budget  for  nsk  mitgation 

DAG  Sec  4.2.33. 

TMPS 

s 

Determine  requrements  for  nsk  mrtigat  on 

DAG  Sec  4.2.33. 

TMPS 

Determine  contractual  changes  for  nsk  mitigation 

OAG  Sec  4.2.33. 

TMPS 

Generate  change  requests  to  mitgate  tecincal  nsk 

INCOSE  Pg  S.10 

TMPS 

s 

Pnontite  handling  of  risks 

INCOSE  PgS.10 

TMPS 

s 

Generate  plan  of  action  for  when  nsk  threshold  rends  acceptable 
levels 

INCOSEPgS.il 

TMP  S 

Utilue  measurements  or  statistics  that  help  manage  the  nsk 

INCOSEPgS.il 

TMPS 

A 

task  Acceptance 

DAG  Sec  4.2.33. 

TMP  S 

A 

Risk  Transference 

DAG  Sec  4.2.33.; 

Suede  Pg  382 

TMP  3 

A 

task  Avoidance 

DAG  Sec  43.33.; 

Bucde  Pg  382 

TMPS 

s 

Avoid  risks  introduced  by  human  error  through  training,  teamwork, 
and  morale 

INCOSE  Pg  7.18 

TMP  S 

s 

Avoid  risks  through  early  procurement,  estate  parallel 
developments,  implement  extensive  analysis  and  testing,  and  make 
contingency  plans 

INCOSE  Pg  7.18 

TMP  S 

3 

Select  most  promising  nsk  handling  option 

INCOSE  Pg  7.18 

100% 

TMP  S 

* 

Beware  of  temptation  to  reduce  verification  activities  due  to  budget  or 
schedule  overruns 

INCOSE  Pg  4.14 

TMPS 

A 

Avoid  conduct**;  verification  late  m  schedule 

INCOSE  Pg  4.14 

TMP  S 

2 

Risk  Modtonng 

DAG  Sec  4.233. 

100%  100% 

IMP'S 

3 

communicate  nsu  to  stakeholders 

DAG  Sec  4333.; 

INCOSE  Pg  S.ll 

100% 

TMPS 

A 

Outknc  risk  reporting  requirements 

DAG  Sec  4.233. 

TMP  S 

A 

Communicate  nsk  management  activities 

DAG  Sec  4.233. 

TMP  $ 

A 

Alert  management  for  plans  implement/ change 

DAG  Sec  4.23  3. 

TMP  S 

3 

Monitor  nsk  mitigation  plans 

DAG  Sec  4.233. 

lDO% 

TMPS 

3 

Review  regular  status  updates 

DAG  Sec  4.2.33. 

100% 

TMPS 

A 

Conduct  periodc  program  management  reviews 

DAG  Sec  4.2.33. 

TMPS 

A 

Conduct  periodc  technical  reviews 

DAG  Sec  4.233. 

TMPS 

3 

Monitor  test  results 

DAG  Sec  4.233. 

100% 

TMPS 

2 

Risk  Documentation 

DAG  Sec  4333.; 
INCOSE  PgS.10 

40%  100% 

TMP  S 

3 

Document  nsk  management  activities 

DAG  Sec  4.233.; 
INCOSE  Pg  S.10 

100% 

TMPS 

3 

Document  change  history 

DAG  Sec  4.233. 

60% 

TMPS 

3 

Document  program  metrics 

DAG  Sec  4.233. 

100% 

TMP  S 

3 

Document  technical  reports 

OAG  Sec  4.2.33. 

100% 

TMPS 

3 

Document  earned  value  reports 

OAG  Sec  4333. 

80% 

TMPS 

3 

Document  watch  lists 

DAG  Sec  4.233. 

40% 

TMPS 

3 

Document  schedjie  performance  reports 

DAG  Sec  4333. 

80% 

TMPS 

3 

Document  technical  review  m.nutes 

DAG  Sec  4.233. 

100% 

TMPS 

3 

Document  critical  nsk  process  reports 

DAG  Sec  4.233. 

100% 

TMPS 

3 

Maintain  record  of  nsk  items  and  how  they  were  handled 

INCOSE  PgS.ll 

100% 

TMP* 

1 

TMP-6  (Configuration  Management! 

S0%  100% 

TMP* 

2 

Develop  Configuration  Baselines 

DAG  Sec  4333.; 
INCOSE  Pg4.l7 

INCOSE  Pg  S.12 

70%  10O% 

TMP* 

3 

Begin  configuration  management  process  m  earty  phases  of  project 

INCOSE  PgS.13 

100% 

TMP* 

3 

Establish  configuration  management  re  sponsor  ties 

INCOSE  Pg  S.12 

Configuration  Control 
Board 

70% 

TMP* 

4 

Identify  gov/ctr  systems  engineering  interaction 

DAG  Sec  4.23  *. 

TMP* 

4 

Identify  gov/ctr  design  engineering  interaction 

DAG  Sec  4.23*. 

TMP* 

4 

Identify  gov/ctr  logistics  interaction 

OAG  Sec  4.2.3  .*. 

TMP* 

4 

Identify  gov/ctr  contracting  interacton 

□AG  Sec  4.23.6. 

TMP* 

4 

Identify  gov/ctr  manufacturing  interaction 

DAG  Sec  4.2.3*. 
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TMP  * 

3 

Establish  structure  for  configuration  identification 

DAG  Sec  4.2.3  *. 

70% 

TMP  * 

3 

Identify  system  elements  to  maintain  under  configuraton  control 

INCOSE  Pg  S.13 

Configuration  Items 

70% 

TMP  6 

4 

Assgn  unique  identifier  to  each  version  of  each  system  clement 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  8.S 

TMP  * 

3 

Establish  architectural  baseline 

INCOSE  Pg  4.H 

100% 

TMP  * 

4 

Ensure  product  functional  characteristics  are  property  identified, 
documented,  validated,  and  verified 

INCOSE  Pg  S.12 

TMP  * 

4 

Ensure  product  performance  characteristics  are  property  identified, 
documented,  validated,  and  verified 

INCOSE  Pg  S.12 

TMP  6 

4 

Document  architectural  design  decisions 

INCOSE  Pg  4.H 

TMP  6 

4 

Obtan  design  approval 

Buede  Pg  31 

TMP  * 

4 

Document  design  approval 

ftjede  Pg  31 

TMP  * 

Provide  substantiated  justification 

INCOSE  Pg  3.7 

TMP  6 

Obtain  approval  of  documentation 

Buede  Pg  40 

TMP* 

3 

Establish  hardware  base^nes 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  S.12 

100% 

TMP  6 

4 

Ensure  product  physical  characteristics  are  property  identified, 
documented,  validated,  and  verified 

INCOSE  Pg  S.12 

TMP  * 

4 

Complete  product  specifications 

INCOSE  Pg  4.10 

TMP  6 

4 

Complete  process  specifications 

INCOSE  Pg  4.10 

TMP  * 

4 

Complete  material  specifications 

INCOSE  Pg  4.10 

TMP  6 

4 

Complete  product  assembly  drawings 

INCOSE  Pg  4.12 

TMP  6 

4 

Complete  manufacturing  tool  drawings 

INCOSE  Pg  4.12 

TMP* 

3 

Establish  software  baselines 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  S.12 

100% 

TMP  * 

2 

Establish  Configuration  Change  Control  Plan  (Establish  configuration  control 
cycle  that  incorporates  evaluation,  approval,  validation,  and  verification  of 
change  requests) 

DAG  Sec  4 .2.3*.; 
INCOSE  Pg  S.13 

so%  so% 

TMP  * 

3 

Establish  charge  request  process 

INCOSE  Pg  S.12 

Configuration  Control 
Board 

so% 

TMP  4 

4 

Ensure  changes  identified 

DAG  Sec  4 .2.3*.. 
INCOSE  Pg  S.12 

TMP  6 

4 

Ensure  changes  recorded 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  S.12 

TMP* 

4 

Ensure  changes  evaluated 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  S.12 

TMP* 

4 

Ensure  charges  ap proved /dsapproved 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  S.12 

TMP* 

4 

Ensure  changes  incorporated 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  S.12 

TMP* 

4 

Ensure  charges  verified 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  S.12 

TMP* 

4 

EstabAsh  configuration  control  board 

INCOSE  Pg  S.13 

TMP* 

3 

Identify  criteria  and  means  for  auditing  element  configuration  to  design 
documentaion 

INCOSE  Pg  4.10 

so% 

TMP* 

2 

Develop  and  Maintain  Configuration  Control  Documentation 

INCOSE  Pg4.b  &  S.13 

so%  so% 

TMP* 

3 

Document  status  and  mpact  of  change  request 

DAG  Sec  4 .2.3*.; 
INCOSE  Pg  S.13 

Change  Request 
Database 

so% 

TMP* 

3 

Identify  and  document  characteristics  of  system  elements  to  be  unque  and 
accessible 

INCOSE  Pg8.S 

so% 

TMP* 

2 

Maintain  Configuration  Baselines 

DAG  Sec  4.2.3*.. 
INCOSE  Pg  4.12  A  S.13 

50%  10O% 

TMP* 

3 

Control  and  maintain  architectural  baselines 

Student  Derived 

100% 

TMP* 

3 

Control,  and  maintain  hardware  base  -its 

INCOSE  Pg  S.12 

80% 

TMP* 

3 

Control  and  maintain  software  baselines 

INCOSE  Pg  S.12 

80% 

TMP* 

3 

Perform  configuration  audits  associated  with  m  destones  and  decison  gates 
to  validate  baselines 

DAG  Sec  4.2.3*.; 
INCOSE  Pg  S.13 

SO% 

TMP* 

4 

Assure  audl  trail  for  decagons 

DAG  Sec  4.2.3*.; 
INCOSE  PgS.S 

TMP* 

4 

Assure  audt  trail  for  design  modfications 

DAG  Sec  4.2.3*.; 
INCOSE  PgS.S 

TMP* 

3 

Update  design  documentation 

INCOSE  Pg  4.10 

100% 

TMP* 

3 

E  ns  ire  consistent  product  versions 

INCOSE  Pg8.S 

100% 
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*MP  7 

1 

TMP-7  (Technical  Data  Management) 

10%  100% 

TMP  7 

2 

Develop  Data  Management  Ran 

DAG  Sec  4.2.3. 7.; 
IhCOSt  Pg  S.1S 

COT  Arorteat^aOata 
Model 

70%  100% 

TMP  7 

3 

Identify  data  management  policies 

DAG  Sec  A. 2. 3-7. 

70% 

TMP  7 

3 

Identify  data  management  systems 

DAG  Sec  4.2.3  7. 

70% 

TMP  7 

3 

Identify  data  management  procedures 

DAG  Sec  A. 2.3. 7. 

70% 

TMP  7 

3 

Identify  method  of  recording  technical  data 

DAG  Sec  4  2.3-7. 

100% 

TMP  7 

4 

Identify  metnod  of  recording  system  development 

DAG  Sec  4.2.3- 7. 

TMP  7 

A 

Identify  method  of  recording  modeling  &  sen 

DAG  Sec  4  2  3-7. 

TMP  7 

A 

Identify  method  of  recording  test  development 

DAG  Sec  4.2.37. 

TMP  7 

A 

Identify  method  of  recording  test  and  evaluation 

DAG  Sec  4.2.37. 

TMP  7 

A 

Identify  method  of  recording  installation 

DAG  See  4.2.37. 

TMP  7 

A 

Identify  method  of  recording  spare  parts 

DAG  Sec  4.2.37. 

TMP  7 

A 

Identify  method  of  recording  repar  parts 

OAG  Sec  4.2.3  7. 

TMP  7 

A 

Identify  method  of  recording  product  sustainment 

DAG  Sec  4.2.3  7. 

TMP  7 

A 

Identify  method  of  recording  supplier  data 

DAG  Sec  4.2.37. 

V- 

A 

Identify  method  of  recording  software  documentation 

DAG  Sec  4.2.37. 

TMP  7 

A 

Identify  method  of  recording  management  info 

OAG  Sec  4.2.37. 

TMP  7 

3 

Establish  and  maintain  data  dictionary 

INCOG*  Pg  S.1S 

70% 

'MP  7 

2 

Determine  /  Define  System  Relevant  Information 

INCOSE  PgS.15 

100%  10 0% 

TMP  7 

3 

Identify  data  ncqs  for  development 

DAG  Sec  4.2.37. 

100% 

TMP  7 

A 

Identify  data  reqs  for  modeling  &  simulation 

OAG  Sec  4.2.3  7. 

TMP  7 

A 

Identify  data  re  as  for  test  development 

OAG  Sec  4.2.37. 

TMP  7 

A 

Identify  data  re  as  for  test  and  evaluation 

OAG  Sec  4.2.37. 

TMP  7 

A 

Identify  data  teas  for  rotalaton 

DAG  Sec  4.2.3  7. 

TMP  7 

A 

Identify  data  re  as  for  spare  parts 

DAG  Sec  4.2.37. 

IMP ; 

A 

Identify  data  re  as  for  repair  parts 

DAG  Sec  4.2.37. 

TMP  7 

A 

Identify  data  re  as  for  product  sustainment 

DAG  See  4.2.37. 

TMP  7 

A 

Identify  data  re  as  for  supplier  data 

DAG  Sec  4.2.3  7. 

TMP  7 

3 

Identify  technical  data  to  be  recorded 

DAG  Sec  4.2.3  7. 

100% 

TMP  7 

A 

Identify  software  documentaton  to  be  recorded 

DAG  Sec  4.2.37. 

IMP-7 

A 

Identify  management  information  to  be  recorded 

OAG  Sec  4.237. 

TMP  7 

A 

Identify  valid  sources  of  information  and  periodcalty  obtain  artifacts  of 
information 

INCOtSlPgS.lS 

TMP  7 

3 

Identify  techncal  data  to  be  communicated 

DAG  Sec  4.2.37. 

100% 

V- 

A 

Identify  software  doc  to  be  communicated 

DAG  Sec  4.2.37. 

TMP  7 

A 

Identify  management  info  to  be  communicated 

DAG  Sec  4.2.37. 

'MP  7 

2 

Identify  System  Data  to  Purchase 

OAG  See  4.2.37.1. 

100%  100% 

TMP  7 

3 

Identify  cost  of  data  delivery 

DAG  Sec  4.2.37.1. 

100% 

TMP  7 

3 

Identify  techncal  data  to  purchase 

OAG  Sec  4.2.37.1. 

100% 

TMP  7 

A 

Identify  circumstances  for  data  to  be  more  useful 

DAG  Sec  4.2.37.1. 

TMP  7 

A 

Identify  circumstances  for  data  to  be  updated 

DAG  See  4.2.37.1. 

TMP  7 

A 

Identify  rcouired  format  of  delivered  data 

DAG  Sec  4.2.37.1. 

TMP  7 

2 

Determine  Data  Protection  Requirements 

OAG  Sec  4.2.37.2. 

100%  100% 

TMP  7 

3 

Identify  if  data  has  restrictions 

DAG  Sec  4.2.3  7.2. 

100% 

TMP  7 

3 

Determine  data  marking  (  release 

DAG  Sec  4.2.3  7.2. 

100% 

TMP  7 

3 

Develop  protection  for  critical  technology  mfo 

OAG  Sec  4.2.37.2. 

100% 

TMP  7 

A 

Reference  50  1 7799  ’Code  of  Practice  for  Information  Security 
Management' 

INCOSt  PgS.lS 

TMP  7 

3 

Establish  (ftstntoution  statements 

DAG  Sec  4.2.3  7.2. 

100% 

TMP  7 

3 

Assure  proper  handling  of  data 

DAG  Sec  4.2.37.2. 

100% 

TMP  7 

A 

Assure  proprietary  data  property  handled 

DAG  Sec  4.2.37.2. 

TMP  7 

A 

Assure  kmited  distribution  data  property  handed 

DAG  Sec  4.2.37.2. 

TMP  7 

A 

Assure  ntellectual  property  data  properly  handled 

DAG  Sec  4.2.37.2. 

TMP  7 

2 

Address  Long  term  Data  Storage  Requirements 

OAG  Sec  4.2.37.3. 

70%  100% 

TMP  7 

3 

Identify  artifacts  for  storage 

INCOS*  Pg  S.1S 

100% 

TMP  7 

A 

Identify  information  rich  artifacts  and  store  for  later  use 

INC  OS*  Pg  S.15 

TMP  7 

3 

Develop  plan  for  digtaing  nformation 

DAG  Sec  4.2.37.3. 

70% 

TMP  7 

A 

Develop  plan  for  dgitued  data  ava>iabikty 

DAG  Sec  4.2.37.3. 

TMP  7 

A 

Develop  plan  for  preserving  digitized  data 

DAG  See  4.2.37.3. 

'mp  ; 

A 

Develop  plan  to  migrate  digit sed  data  to  new  form 

DAG  Sec  4.2.37.3. 

TMP  7 

3 

Address  retrieval  of  data 

DAG  Sec  4.2.3  7.3. 

100% 

TMP  7 

2 

Record  Program  Data 

INCOG*  Pg  4.10 

10%  100% 

TMP  7 

3 

Complete  specifications 

INCOS*  PR  4.10 

80% 

TMP  7 

A 

Complete  product  specifications 

INC  OS*  Pr  4.10 

TMP  7 

A 

Complete  process  specifications 

INCOS*  Pr  4.10 
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TMP  7 

4 

Complete  material  specifications 

INCOSE  Pg  4.10 

TMP  7 

3 

Update  design  documentation 

incose  pg  4.xo 

100% 

TMP  7 

3 

Document  testng  and  analysis  resists 

ftjede  Pg  42 

100% 

TMP  7 

4 

Document  verification  testing 

Buede  Pg  42 

TMP  7 

A 

Document  validation  testing 

Rjede  Pg  42 

TMP  7 

A 

Document  ntegration  testing  results 

INC  OSS  Pg  4.12 

TMP  7 
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